Method for scanning dental arch of user

Abstract

The invention relates to a method for scanning a dental arch of a user, comprising:-a pre-step of generating a reference representation, and then-performing a plurality of cycles of the following steps a) to d): a) acquiring an updated image by the user by means of an image acquisition device; b) analyzing the updated image in order to identify a surface of the updated image representing the dental arch of the user, referred to as an "acquired 2D surface", then delineating and marking a surface of the reference representation corresponding to the acquired 2D surface, or referred to as a "basic marking area", and adding the basic marking area to the "marking area of the reference representation", the mark area represented by the reference gathers all the basic mark areas; c) determining and presenting guidance information for guiding towards the new acquisition condition; d) preferably moving the acquisition device towards the new acquisition condition.

Description

Technical Field

[0001] The invention relates to a method for scanning a dental arch of a user and to a device for implementing such a method.

[0002] Prior Art

[0003] Traditionally, the dental arch of a user is scanned in order to analyze the dental condition, especially before establishing orthodontic treatment. However, scanning the dental arch of a user requires the user to visit a dental care professional or to transmit information, especially images, to a dental care professional. However, scanning in a dental office, or more generally from a dental care professional, may not only be costly but also stressful for the user. Furthermore, dental images acquired by the user themselves may result in insufficient image quality or incomplete content.

[0004] There is therefore a need to facilitate the acquisition of dental arch scans while limiting the risk of incomplete scans.

[0005] It is an object of the present invention to meet this need. Summary of the invention Summary of the invention

[0007] The present invention provides a method for scanning a dental arch of a user, the method comprising:

[0008] - A preliminary step of determining a reference dental arch model called a "reference representation", then

[0009] - Perform multiple cycles of the following steps a) to d):

[0010] a) preferably by means of an image acquisition device, by a user, acquiring a two-dimensional image of the dental arch under acquisition conditions, or an "updated image";

[0011] b) analyzing the updated image in order to identify a surface of the updated image representing the dental arch of the user, called the “acquired 2D surface”, then outlining and marking a surface of the reference representation corresponding to the acquired 2D surface, called the “basic marking area”, and adding this basic marking area to the “marked area of ​​the reference representation”, which brings together all these basic marking areas defined during the previous cycles, which were empty before the first cycle in said cycles;

[0012] c) determining and presenting guidance information to a user for guiding the acquisition device toward a new acquisition condition suitable for a next cycle, thereby increasing the marked area of ​​the reference representation;

[0013] d) moving the acquisition device towards said new acquisition conditions; the method preferably comprises, after the last cycle, a final step of generating a final three-dimensional model from said updated images.

[0014] The generation of the final model may be performed by the image acquisition device or by a computer remote from the image acquisition device and to which the set of acquired updated images has been transferred, preferably to the computer of a dental care professional.

[0015] As will be seen in more detail later in the description, the scanning method according to the invention quickly and easily produces a set of updated images sufficient to produce a complete final model of the dental arch.

[0016] Furthermore, the user does not need to visit a dental care professional, or the user does not need to be supervised by a dental care professional to acquire the update image. Through guidance, the update image can be advantageously acquired under precise acquisition conditions without requiring any special training.

[0017] Advantageously, the acquisition may be performed in real time, for example, using updated images extracted from the video.

[0018] Preferably, in step c), the guidance information is presented on a screen, preferably on a screen of an image acquisition device. In particular, the image acquisition device may be a cellular phone or a tablet computer, preferably a cellular phone.

[0019] In particular, the guidance information may include audio, visual and / or sensory information. In this way, the guidance information can be customized for any user. Preferably, the guidance information includes several different types of information, preferably stimulating several different senses, thereby promoting communication with the user.

[0020] In a preferred embodiment of the invention, for each step b), the marking comprises visual markings of the surface represented by the reference, and each step c) comprises presenting all visual markings generated in all previous steps b), preferably to the user on a screen, preferably on the screen of a user's cellular phone used to capture the updated image.

[0021] The visual marking preferably comprises a graphical representation of the primary marking area, such as a representation of an outline of the primary marking area and / or a solid color and / or pattern within the borders of the primary marking area.

[0022] This simplifies acquisition. In particular, during real-time looping, the screen displays a reference representation of the marked areas as the loop progresses, enabling the user to easily identify unmarked areas and position and orient the image acquisition device accordingly.

[0023] Instead, during a real-time cycle, the display may show a reference representation in which the unmarked areas decay as the cycle progresses. Preferably, however, the screen continues to represent at least the outline of the dental arch.

[0024] Therefore, visual markers are the preferred form of guidance information.

[0025] The guidance information may include additional information corresponding to information other than the information intended to indicate the positioning of the acquisition device, which may, for example, instruct the user to insert or remove an orthodontic appliance, open or close the mouth, and / or use a retractor or conversely remove it.

[0026] A reference representation may be generated. Alternatively, it may be selected, for example, from a database.

[0027] The reference representation may be two-dimensional. In a preferred embodiment, the reference representation is a digital three-dimensional model. By acquiring different update images, the method annotates different areas of the model until a model coverage threshold is reached, or preferably until the model is completely labeled. The update images then advantageously "cover" a large portion of the model surface, or preferably the entire model surface, and enable, for example, a final model representing the user's dental arch to be generated with high accuracy. The final model may be generated by a dental care professional's computer to which the update images are transmitted, or by an image acquisition device.

[0028] In particular, the coverage threshold may be 90%, better 95%, even better 99%.

[0029] At the end of each cycle, the method may include comparing the percentage of the surface area of ​​the marked region to the total surface area of ​​the model (called the "coverage percentage") with a coverage threshold. If the coverage percentage is above the threshold, the final model generation step is performed, otherwise a new cycle is started.

[0030] In certain embodiments, in step c), the coverage percentage is presented to the user.

[0031] The presentation of coverage percentages may take the form of a meter, a numerical display, a progress bar, a goal that is presented continuously when a particular goal is reached, or a simultaneous presentation of different displays that distinguish the goals that have been reached from those that remain to be reached. For example, the goals are "Capture molars", "Capture incisors", "Capture more than 10% of model", "Capture more than 25% of model", "Capture more than 40% of model", "Capture more than 50% of model", "Capture more than 75% of model", "Capture more than 85% of model", "Capture right view", "Capture left view", "Capture top view", "Capture bottom view"; "Capture mouth closed view", "Capture mouth open view", "Capture with orthodontic appliances".

[0032] The method according to the invention may also include in particular one or more of the following optional features:

[0033] - the continuously looped updated images are extracted from a video captured by the user via a tablet or a cell phone, said looping preferably being performed in real time during the capture of said video, at least part of the guidance information preferably being displayed on the screen of said tablet or said phone, respectively;

[0034] - the guidance information comprises a marked area presented on a screen of the image acquisition device, preferably superimposed on the reference representation, wherein the unmarked area has a different appearance, preferably a color, than the marked area;

[0035] - The reference representation is the "reference" model of the "reference" dental arch

[0036] - means a group of "historical" dental arches comprising more than 10, more than 100, more than 1,000, more than 10,000 and / or less than 1,000,000 historical individuals, or

[0037] - by assembling the dental model in an arrangement determined by analysis of one or more analysis images of the user's dental arch, preferably acquired with an image acquisition device

[0038] - one or more 3D or 2D illustrations of the dental arch, which may be photorealistic;

[0039] - in step c) of any cycle, presenting a view of the observed reference model along a viewing direction oriented relative to the reference model, which viewing direction coincides with a direction of acquisition of the image updated in step a) of said cycle relative to the dental arch of the user, this acquisition direction being oriented substantially coincident with the direction of the optical axis of the camera during said acquisition, so that said view can be superimposed substantially in alignment with said updated image;

[0040] - in step c), preferably in any step c), the image acquisition device provides variable information, for example by emitting an acoustic signal or a visual indication (for example a flashing light), depending on the difference between the actual acquisition conditions and said new acquisition conditions, the greater said difference is; the user thus advantageously knows whether the movement of the image acquisition device he has performed is more or less effective in reaching the new acquisition conditions;

[0041] - for at least one cycle of step a), preferably for any step a), the lips and / or cheeks of the user's dental arch are spread apart, preferably using a retractor, in order to expose at least one tooth to the image acquisition device, for example to fully expose the outer surface of the incisors and / or at least partially expose the outer surface of the molars;

[0042] -During the cycle, the following are collected:

[0043] - at least one updated image taken facing the user and / or at least one updated image taken to the right of the user and / or at least one updated image taken to the left of the user, preferably at least one updated image taken facing the user and at least one updated image taken to the right of the user and at least one updated image taken to the left of the user; and / or

[0044] - at least one update image taken facing the user on the one hand and at least one update image taken from above the user and / or at least one update image taken from below the user on the other hand; preferably at least one update image taken facing the user and at least one update image taken from above the user and at least one update image taken from below the user; and / or

[0045] - at least one updated image taken with the mouth open and / or at least one updated image taken with the mouth closed;

[0046] - the user is at a distance preferably more than 100 m or more than 1 km and / or less than 1000 km from any dental care professional (in particular any orthodontist);

[0047] - stopping the loop when more than 50%, more than 70%, more than 90%, preferably 100% of the surface of the reference representation is "covered" by the acquired update images depicted in at least one update image;

[0048] - comparing the final model with another model representing the dental arch of the user, for example to correct the final model or the other model and / or to decompose the final model into a tooth model and preferably to move the tooth model;

[0049] - comparing the final model with a reference model and preferably presenting the comparison to a user, for example by superimposing the final model and the reference model, preferably in an aligned manner;

[0050] - The reference representation is a three-dimensional model and the surface of the reference representation corresponding to the acquired 2D surface is determined by:

[0051] - by searching for suitable virtual acquisition conditions for acquiring views of a reference representation, called "reference image", and which best matches the image updated under said virtual acquisition conditions; then

[0052] - by projecting the acquired 2D surface onto a reference representation, along a viewing direction of said reference representation under said virtual acquisition conditions;

[0053] - The virtual acquisition conditions are determined by seeking to superimpose the salient points of the dental arch represented on the updated image with the corresponding salient points of the reference representation.

[0054] The present invention also relates to:

[0055] - a computer program, in particular dedicated software for a cell phone or tablet or an "app", comprising program code instructions for carrying out one or more of the steps of the method according to the invention, preferably for all steps b) to c) when said program is executed by a computer,

[0056] - a data medium on which such a program is recorded, for example a memory or a CD-ROM.

[0057] Preferably, when the computer program is dedicated software for a cell phone or tablet computer, the computer program comprises instructions for performing step a), and even more preferably instructions for performing all steps a) to c).

[0058] The invention also relates to a device for implementing the method according to the invention, comprising:

[0059] - Optionally, a first computer having a first computer program comprising program code instructions for generating a reference representation or selecting a reference representation, e.g. from a database;

[0060] - an image acquisition device for performing step a);

[0061] - preferably, a screen for displaying guidance information during step c);

[0062] - a second computer, identical to or different from the first computer, the second computer comprising a second computer program comprising program code instructions for performing steps b) and c);

[0063] - a third computer, identical to or different from the first computer and / or the second computer, the third computer comprising a third computer program comprising program code instructions for implementing the final step of generating a final model of the user's dental arch from the updated image;

[0064] - Preferably, a fourth computer, identical or different to the first computer and / or the second computer and / or the third computer, comprising a fourth computer program comprising program code instructions for comparing the final model with another model representing a dental arch of the user and / or for decomposing the final model into tooth models and preferably moving said tooth models.

[0065] Preferably, the screen and / or the first computer and / or the second computer and / or the third computer and / or the fourth computer are integrated into the acquisition device. Preferably, the image acquisition device is a cellular phone or a tablet computer. Preferably, the screen is a cellular phone or a tablet computer screen, respectively.

[0066] The user can then acquire good quality dental images that together fully represent the reference representation, without third party intervention, and in particular without intervention by a dental care professional, with the aid of a simple cell phone or tablet.

[0067] The device may further comprise communication means, in particular for sending the update image and / or receiving the reference representation.

[0068] definition

[0069] A "user" is a person on whom the method according to the invention is carried out.

[0070] The term "dental care professional" refers to anyone qualified to provide dental care, including orthodontists and dentists.

[0071] The "mouth closed" position is a bite position in which the user's upper and lower teeth are in contact. The "mouth open" position is a position in which the user's upper and lower teeth are not in contact, preferably a position in which the mouth is fully open.

[0072] "Dental arch" refers to all or part of a dental arch, preferably comprising at least 2 teeth, preferably at least 3 teeth, more preferably at least 4 teeth.

[0073] A retractor (or dental retractor) is a device for pulling the lips back. The retractor includes upper and lower flanges and / or right and left flanges extending around the retractor opening and intended to be inserted between the teeth and the lips. In the operating position, the user's lips rest on these edges so that the teeth are visible through the retractor opening. Thus, the retractor allows the teeth to be viewed without being obstructed by the lips. However, the teeth do not rest on the retractor so that by turning the head relative to the retractor, the user can change the teeth visible through the retractor opening. The user can also change the spacing between their dental arches. In particular, the retractor does not press on the teeth to expand the upper and lower jaws apart, but presses on the lips. In one embodiment, the retractor is configured to elastically expand the upper and lower lips apart to expose the teeth visible through the retractor opening. In one embodiment, the retractor is configured so that the distance between the top edge and the bottom edge and / or between the right edge and the left edge is constant. Retractors are described, for example, in PCT / EP2015 / 074896, US 6,923,761 or US 2004 / 0209225.

[0074] According to the international practice of the FDI World Dental Federation, each tooth in the dental arch has a predetermined number.

[0075] A "notable point" is a point on the dental arch, tooth, tooth model or tooth model that can be identified, such as the apex of a tooth or the tip of a cusp, an interdental (i.e., a tooth with an adjacent tooth) contact point (such as the mesial or distal point of the incisal edge of a tooth), or a point at the center of a crown, or a "center of gravity."

[0076] "Computer" refers to a computer processing unit, which includes a group of machines with computer processing capabilities. In particular, the unit may be integrated into a cell phone, or may be a PC-type computer or a server, such as a server remote from the user, such as a computer in the "cloud" or located at the dental care professional's premises. The cell phone and computer then have components that communicate with each other.

[0077] Typically, a computer includes a processor, a memory, a human-computer interface (usually including a screen), and a computer that can communicate with the Internet, WIFI, or a communication module of a telephone network. Software configured to implement the method of the present invention is loaded into the memory of the computer. The computer may also be connected to a printer.

[0078] The method according to the invention (except the operations of acquiring and moving the image acquisition device) is implemented by a computer, preferably only by a computer.

[0079] The term "model" refers to a three-dimensional digital model. A model is made up of a set of voxels.

[0080] A "tooth model" is a three-dimensional digital model of teeth. The dental arch model can be cut to define a tooth model of at least some (preferably all) of the teeth represented in the dental arch model. Thus, the tooth model is a model within the dental arch model.

[0081] A "dental arch model" is a model representing at least a portion of a dental arch, preferably at least 2 teeth, preferably at least 3 teeth, most preferably at least 4 teeth.

[0082] "Decomposing" the dental arch model into "tooth models" is the operation of delimiting and making autonomous the tooth representations (tooth models) in the dental arch model. Computer tools can be used to manipulate the tooth models in the dental arch model. An example of software for manipulating the tooth model and creating a treatment plan is the program Treat described at https: / / en.wikipedia.org / wiki / Clear_aligners#cite_note-invisalignsystem-10.

[0083] An "image" is a two-dimensional picture, such as a photo or a video frame. An image is made up of pixels. A "video" is considered a collection of photos.

[0084] A "dental arch image" or a "dental arch model" or a "reference representation" of a dental arch refers to a representation of all or part of said dental arch, preferably representing at least 2 teeth, more preferably representing at least 4 teeth.

[0085] The "match" or "fit" between two objects (e.g., between two images of a dental arch) is a measure of the difference or "distance" between them. A "best fit" is achieved when this difference is minimal, particularly when the two images represent the same elements in substantially the same way, i.e., in such a way that the representations of the elements on the two images are substantially superimposable and aligned.

[0086] When the outline of the representation of a physical element in a first image is superimposed on the outline of said physical element in a second image, the superposition of the two images is "aligned." Thus, the superposition appears to be real.

[0087] "Marking" a surface with a reference representation includes storing sufficient data in a computer memory to specifically reconstruct the surface. In particular, the boundaries of the surface may be represented on the reference representation so that the user can visualize it. The marking is then described as "visual".

[0088] The "acquisition conditions" of an image specify the spatial position and orientation of the device used to acquire the image relative to the user's teeth (actual acquisition conditions) or relative to a model of the user's teeth (virtual acquisition conditions), and preferably specify the calibration of the acquisition device. Acquisition conditions are called "virtual" or "theoretical" when they correspond to a simulation in which the acquisition device is in said acquisition conditions relative to the model (theoretical positioning and preferred calibration of the acquisition device).

[0089] In particular, in the case of a virtual acquisition of a reference image, the acquisition device can also be described as "virtual". The image is actually acquired by a virtual acquisition device which has the properties of a "real" acquisition device for acquiring a real image, in particular an update image.

[0090] The "calibration" of an acquisition device consists of all calibration parameter values. Calibration parameters are parameters intrinsic to the acquisition device (other than its position and orientation), whose values ​​influence the acquired images. Preferably, the calibration parameters are selected from the group formed by aperture, exposure time, focus and sensitivity.

[0091] "Statistical processing" when applied to a data set enables us to determine a characteristic property of that set, such as the mean, standard deviation or median. Statistical processing algorithms are well known to those skilled in the art.

[0092] A "diagram" has no physical reality. It may be a symbolic or schematic representation comprising a set of geometric shapes, or a 2D or 3D digital drawing.

[0093] For the sake of clarity, the terms “first,” “second,” “third,” and “fourth,” “reference,” “update,” “historical,” and “goal” are used.

[0094] “Vertical”, “horizontal”, “right”, “left”, “in front” or “from the front”, “behind”, “above”, “below” mean that the user is standing vertically.

[0095] Unless otherwise indicated, "including" or "comprising" or "having" should be interpreted in a non-limiting manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Other features and advantages of the present invention will become apparent from the following detailed description and from an examination of the accompanying drawings, in which:

[0097] [ Figure 1 ] Figure 1 An example of a cycle of the method according to the invention is schematically shown;

[0098] [ Figure 2 ] Figure 2 An example of an apparatus according to the invention is schematically shown;

[0099] [ Figure 3 ] Figure 3 A progressive coloring of the reference model is shown. DETAILED DESCRIPTION

[0100] method

[0101] The method of scanning a user's dental arch according to the present invention comprises: Figure 1 Steps a) to d) are shown.

[0102] Preferably, the method further comprises a preliminary step of determining a reference representation.

[0103] Determining a reference representation may involve generating a representation, in particular a three-dimensional model.

[0104] In another embodiment, determining the reference representation may include selecting a representation from a database.

[0105] The reference representation is a model or an image representing a reference dental arch.Preferably, the reference representation is a model referred to as a "reference model".

[0106] The reference representation may represent a standard reference dental arch and may be used for different users, possibly for any user. However, the reference representation preferably represents a reference dental arch having a shape close to that of the user, which improves the accuracy of the method.

[0107] The reference representation may represent a population of individuals.

[0108] In one embodiment, the reference representation is obtained by statistically processing a collection of historical dental arch representations of a historical group of individuals having one or more characteristics in common with the user, such as grouping together people who are in the same age group as the user or who suffer from the same condition as the user. The statistical processing may, for example, include determining a reference representation that minimizes shape differences from the historical dental arch representations.

[0109] In particular, the reference representation may be a dental cast model.

[0110] The reference representation can be obtained by arranging the historical dental model in a manner corresponding to the arrangement of one or more "analysis" images of the user's dental arch, preferably taken with the user's cell phone. In particular, this can be done as follows:

[0111] - creating a historical library of more than 1,000, preferably more than 10,000 and / or less than 1,000,000 tooth models (so-called "historical tooth models") and preferably assigning a tooth number to each historical tooth model;

[0112] - analyzing at least one "analysis image" of the dental arch, for example by means of a deep learning device, preferably by means of a neural network, in order to determine at least one region of the analysis image representing teeth, or a "teeth analysis region", and preferably the number of said teeth;

[0113] - for each tooth analysis region determined in the previous step, searching the history library for a historical tooth model that has the greatest closeness in shape to the tooth analysis region, or the "optimal tooth model", and preferably has the same number of teeth as the teeth depicted in the tooth analysis region;

[0114] - arranging the set of optimal dental models to create a dental arch model that has the greatest approximation to the dental arch depicted in the analysis image, or "assembled model";

[0115] - optionally, replacing at least one optimal tooth model with another tooth model and repeating the previous steps in order to maximize the match between the assembled model and the analyzed image;

[0116] - Optionally, repeating the analysis with another analysis image and, in a step of permutation and / or replacement, searching for the best fit for the set of analysis images used.

[0117] A method for generating a reference representation in the form of a model by arranging historical dental models is described in detail in European application No. 18 184486.

[0118] The reference representation may be generated by an image acquisition device, in particular from an image of a user's dental arch taken with the image acquisition device.The reference representation may be generated by another device, such as a computer of a dental care professional.

[0119] The reference representation may be stored in a memory or cloud accessible via digital communication means.The reference representation may be accessed by a computer performing steps b) and c).

[0120] After providing the reference representation, the method comprises a plurality of cycles of steps a) to d).

[0121] The time interval between two consecutive cycles is preferably less than 5 minutes, 1 minute, 30 seconds or 1 second. Preferably, the user acquires the update images in real time, preferably by taking a video of his dental arch, and steps b) to d are performed immediately for each update image. The reference representation is thus marked in real time, preferably visually, and the user is guided through the process of making the video.

[0122] Preferably, the loop is performed for at least one updated image taken facing the user, at least one updated image taken to the right of the user, and at least one updated image taken to the left of the user.

[0123] Preferably, the loop is performed for at least one updated image taken when the mouth is open and for at least one updated image taken when the mouth is closed.

[0124] In a first cycle, the reference representation is partially labeled in step b). The reference representation labeling is then completed for each subsequent cycle.

[0125] An example cycle is now described in detail.

[0126] In step a), an update image, preferably a photograph, depicting the dental arch of the user is captured using an image capture device. Preferably, a video is captured using the image capture device and the update image is extracted from the video.

[0127] In a particular embodiment, step a) comprises acquiring, by means of an image acquisition device, a plurality of updated images, preferably photographs, representing the dental arch of the user, which plurality of images may be extracted from a video.

[0128] Preferably, the acquisition is performed by the users themselves.

[0129] The updating image is preferably "extra-oral", ie without inserting the camera lens into the user's mouth.

[0130] The image acquisition device can in particular be a cellular phone, a tablet, a camera or a computer, the image acquisition device preferably being a cellular phone or a tablet, in particular so that the user can acquire up-to-date images anywhere, and in particular outside the dental care professional's office, for example more than 1 km away from the dental care professional's office.

[0131] In one embodiment, the user uses a cell phone and a holder to which the cell phone can be removably attached, which is held against the user during the acquisition of any update images. In particular, the holder can be of the type described in PCT / EP2021 / 068702, EP17306361, PCT / EP2019 / 079565, PCT / EP2022 / 053847 or FR2113577.

[0132] In a preferred embodiment, the user uses a free cell phone, i.e. a cell phone whose position and orientation can be determined freely, in particular a cell phone that is not attached to a holder. In fact, the method described in the invention enables the user to be guided when taking the image, so that guidance by means of a holder is not necessary. Preferably, the image acquisition device is not in contact with the user's mouth, either directly or via a holder of the image acquisition device.

[0133] The time interval between the preliminary step of generating the reference representation and step a) is preferably less than 4, 2 or 1 week, in particular when said generation is performed based on an analysis image of the user's dental arch. The preliminary step may also be performed immediately before step a), for example less than 5 minutes before step a).

[0134] In step b), the updated image is analyzed to identify in the updated image a surface at least partially depicting the dental arch of the user, called the "captured 2D surface".

[0135] When a plurality of update images are acquired in step a) prior to step b), each update image of the plurality of update images is analyzed to identify the acquired 2D surface in each update image.

[0136] The analysis can be performed using conventional segmentation methods.

[0137] A search is then performed for a portion of the surface of the reference representation corresponding to the acquired 2D surface. Preferably, the portion results from a one-to-one relationship with the acquired 2D surface, ie for each acquired 2D surface a single portion of the surface of the reference representation is defined.

[0138] Preferably, the portion is the result of a projection of the acquired 2D surface onto a reference representation.

[0139] In a preferred embodiment, the reference representation is a model. We then search for a model view that matches the update image, which model view represents the modeled dental arch in substantially the same way as the update image. In other words, virtual acquisition conditions are sought that are suitable for acquiring a model view that best fits the image updated under said virtual acquisition conditions. These virtual acquisition conditions can be determined by searching for virtual acquisition conditions that superimpose salient points on the update image onto corresponding salient points on the model, preferably with the aid of an optimization algorithm.

[0140] The search for a model view matching an update image is described, for example, in European application no. 18 184486.

[0141] The view and update image can then be superimposed substantially aligned with each other. By superimposing the model view and update image in alignment, a view region covered by the acquired 2D surface can be defined. The boundaries of this region can be transferred to the model. Once completed, they outline a portion of the reference representation surface corresponding to the acquired 2D surface. The portion is then the result of projecting the acquired 2D surface onto the reference representation along the model viewing direction of the view representation model.

[0142] Then, the portion of the reference representation surface corresponding to the acquired 2D surface is marked. This is called the "basic marked area". The "marked area" is the union of all basic marked areas defined during the cycle. The marked area, which is initially empty before the first cycle, therefore includes the basic marked area at the end of the first cycle, and is then completed at each cycle as the acquisition device moves and updated images are acquired. Therefore, the "unmarked area" representing the portions of the reference representation corresponding to the areas of the dental arch that do not appear on any updated image gradually decreases.

[0143] In step c), image acquisition device guidance information is determined and presented.

[0144] The guidance information is useful for helping the user to move the image acquisition device so that during the next cycle, the acquired updated image represents a newly acquired 2D surface that enables completion of the marking area, i.e., this results in the basic marking area being at least partially outside the marking area defined at the end of step b) of the current cycle.

[0145] The guidance information informs the user which parts of their dental arch still require acquisition of one or more updated images. It guides the user to orient and / or position the acquisition device to achieve the desired acquisition.

[0146] In one embodiment, the computer implementing step c) determines the new acquisition conditions for the next cycle, such as by random search or using an optimization algorithm, so that the new 2D surface acquired under these new acquisition conditions maximizes the increase in the marked area of ​​the reference representation. These new acquisition conditions can be described as new "target" acquisition conditions.

[0147] The guidance information may include visual information and / or auditory information and / or sensitive information. For example, the guidance information may include audio instructions that tell the user to move the collection device toward or away from the teeth, move the collection device to the right or left, rotate the collection device around the dental arch, open or close the mouth, or open or close the jaw.

[0148] Sensitive information could be a vibration, for example, indicating to the user to stop moving.

[0149] Preferably, in step c), the visual guidance information is presented, preferably on a screen, preferably on a screen of an image acquisition device, preferably on a cell phone or tablet screen.

[0150] In one embodiment, marked areas and preferably unmarked areas are displayed. This display helps to guide the user quickly and efficiently. This guidance system gives the user a lot of freedom, is intuitive, and therefore does not require prior training.

[0151] The real-time loop enables immediate viewing of how the marked area is extended. The gradual coloring of the reference representation is pleasing. In one embodiment, a goal may be set for the user, for example, to encourage them to color the reference representation as quickly as possible, further enhancing the pleasurable aspect of acquiring updated images.

[0152] Preferably, the marked area is colored, preferably a different color from the unmarked area. Preferably, the outline of the basic marked area is not specifically indicated, and only the outline of the marked area may be shown. Then, preferably, the marking in step b) is a visual marking, including extending the colored marked area to merge it into the basic marked area defined in said step b).

[0153] By coloring the marked areas, and preferably by displaying the unmarked areas, the user is indirectly instructed how to move the image acquisition device so as to acquire during the next cycle an updated image representing a portion of the dental arch corresponding to the uncolored areas and therefore to the unmarked areas of the reference representation.

[0154] In the same way as the progressive coloring of the marked areas, the unmarked areas can be progressively erased. On the screen, the two solutions are similar, the deletion of an area causes it to be colored with the background color.

[0155] The guide information may be determined according to the acquisition conditions of the update image acquired in the previous step a) and the unmarked area.

[0156] Determining guidance information from unmarked areas, or equivalently from marked areas, means that the guidance information does not depend solely on the position of the image acquisition device in space. It depends on what the image acquisition device has observed up to that point, and therefore depends not only on the position of the image acquisition device in space, but also on the orientation of the image acquisition device lens relative to the dental arch during image acquisition.

[0157] As described in US2020193871A1, the representation of the trajectory of the image acquisition device does not depend on the marked area, but only on the position of the image acquisition device in space.

[0158] Steps b) and c) are implemented by a computer program.

[0159] In step d), the user modifies the position and / or orientation of the acquisition device according to the guidance information.

[0160] Preferably, the cycle of steps a) to d) is repeated until a stop criterion is reached.

[0161] The stopping criterion may be the ratio of the marked surface area to the surface area of ​​the reference representation. In particular, steps a) to d) may be repeated until the ratio is greater than 50%, preferably greater than 75%, more preferably greater than 80%, more preferably greater than 90%, more preferably greater than 95%, more preferably 100%, i.e. until the reference representation no longer contains any unmarked areas.

[0162] Finally, the scanning method according to the invention comprises, after stopping the cycle from a) to d), a step for generating a 3D model from the updated images acquired during the respective cycle.

[0163] Preferably, after cycling through steps a) to d), the method comprises the step of sending updated images acquired during each cycle and / or models generated from updated images acquired during each cycle, preferably to a dental care professional.

[0164] In particular, the model may be used by a dental care professional for diagnosis or designing an orthodontic treatment plan, or even for manufacturing orthodontic appliances, preferably orthodontic braces.

[0165] Device

[0166] The invention also relates to a device for implementing the scanning method according to the invention. Figure 2 Shown in.

[0167] The device 1 according to the invention comprises:

[0168] - Optionally, a first computer comprising a first computer program comprising program code instructions for generating a reference representation;

[0169] an image acquisition device 10 for performing step a) to acquire an update image 12,

[0170] a display 14 for presenting guidance information 16 during step c),

[0171] a second computer 18 having a second computer program comprising program code instructions for executing steps b) and c), the second computer 18 receiving the reference model 20 stored in the database 22,

[0172] - preferably a third computer, which is the same as or different from the first computer and / or the second computer, the third computer comprising a third computer program comprising program code instructions for generating a final model of the user's dental arch from the updated image;

[0173] - Preferably, a fourth computer, identical or different to the first computer and / or the second computer and / or the third computer, comprising a fourth computer program comprising program code instructions for comparing the final model with another model representing a dental arch of the user and / or for decomposing the final model into tooth models and preferably moving said tooth models.

[0174] Preferably, the screen and / or the first computer and / or the second computer and / or the third computer and / or the fourth computer are integrated into the image acquisition device.

[0175] Preferably, the image acquisition device is a cellular phone or a tablet computer with a camera and a screen, wherein the first computer program and / or the second computer program and / or the third computer program and / or the fourth computer program are preferably loaded into the cellular phone or the tablet computer in the form of dedicated software.

[0176] The apparatus may comprise communication means for sending an update image and / or video when the update image is extracted from the video and / or for sending a final model generated from the update image at the end of a cycle of steps a) to d) and / or for receiving a reference representation prior to the first cycle.

[0177] The apparatus may include a memory for storing the update image and / or the video when the update image is extracted from the video, and / or for storing the final model generated from the update image.

[0178] Example

[0179] The user wishes to obtain a model of at least one of his dental arches without having to visit a dental care professional, for example to check the state of his teeth. The user launches a dedicated software program or "app" on his cell phone or tablet that enables it to implement the method according to the invention.

[0180] Special-purpose software is considered to have performed a step if it performs the step itself, or communicates with a remote computer for that purpose.

[0181] In a preliminary step, the dedicated software may ask the user to acquire analysis images, preferably to ask him to take some photographs or videos, in order to determine the characteristics of one of his dental arches and to select a reference model in the form of a reference representation from a database, which represents a reference dental arch with the same or similar characteristics. In particular, the analysis of photographs, possibly extracted from a video, can be used to determine the general shape of the dental arch. For example, the reference model can be a dental cast model with the same width as the user's dental arch.

[0182] Alternatively, the reference model may be a standard dental cast model available for a class of users, or even a generic dental cast model available for any user.

[0183] Specialized software then displays the reference model on the cell phone screen and asks the user to take a video of the dental arch.

[0184] In step a), in selfie mode and preferably without using a retainer, the user takes a video of his dental arch. Preferably, however, the user uses a retractor and in particular expands the corners of the mouth to take a picture of the teeth at the back of his mouth and / or expands the upper and lower lips to expose the front teeth, preferably the gums around the front teeth. The update images are stills taken from the video, preferably at regular time intervals.

[0185] In step b), each updated video image is analyzed by dedicated software to identify whether it represents a portion of a dental arch. If necessary, this analysis can identify a portion of the updated image, possibly several segments, which represents a dental arch, ie, identify a "captured 2D surface".

[0186] The acquired 2D surface is then projected onto the reference model. To this end, the dedicated software determines the direction in which to view the reference model so as to obtain a view that is substantially identical to the representation of the user's dental arch in the updated image. The acquired 2D surface is then projected onto the reference model in that direction, thereby identifying portions of the reference model surface that represent portions of the reference dental arch that are identical to those portions of the user's dental arch represented by the acquired 2D surface.

[0187] This portion of the reference model surface, or "marked elementary region", is then added to the marked regions of the reference model, which are the union of all elementary regions marked in the previous cycle. In this way, the dedicated software can indicate at any time an unmarked region for which at least one updated image has not yet been taken and analyzed.

[0188] In this example, it is assumed that the goal is to cover the entire reference model, i.e., the marked area consists of the entire surface of the reference model, e.g. Figure 3 tf in FIG. 1 . However, in one embodiment, the method aims to overlay only a portion of the surface of the reference model, and more generally a portion of the reference representation. The portion may be identified on the reference model, for example by outlining it. Alternatively, only the portion may be displayed on a cell phone or tablet screen.

[0189] Specialized software then provides guidance so that the user can reach the target using the camera on a cell phone or tablet.

[0190] In a preferred embodiment, dedicated software visually annotates the marked areas, preferably by coloring them differently from unmarked areas. The user immediately sees the unmarked areas and positions the camera accordingly. Since the coloring is real-time, the user receives instant feedback on the relevance of the instantaneous positioning of the camera and can adjust it accordingly.

[0191] Figure 3 Different views of a reference model are shown, characterized by the evolution over time t of the visual annotations constituting the guidance information, the reference model 20 consisting of a tooth model 30 progressively colored according to image acquisitions made during a cycle executing steps a) to d).

[0192] Before the first cycle, at time t0, the labeled region of the reference model is an empty set.

[0193] Each tooth model 30 can be colored when more than 80% of the surface of the tooth model is covered by the marked area of ​​the reference model, preferably when more than 90% of the surface of the tooth model is covered by the marked area of ​​the reference model, and even better when the entire surface of the tooth model is covered by the marked area of ​​the reference model. The presentation of the colored tooth model 32 provides guidance information for the user.

[0194] The tooth model may be partially colored, the coloring corresponding to the surface of the tooth model covered by the marked area of ​​the reference model. In this way, the user can easily orient the image acquisition device to acquire the surface of the tooth model not covered by the marked area of ​​the reference model. Alternatively or preferably, in addition to the visual markings, other guiding information may be provided.

[0195] In one embodiment, the dedicated software determines a new "target" acquisition condition for the camera, i.e., an acquisition condition that will, for example, result in the largest acquired 2D surface or the largest element marking area, i.e., a new acquisition condition that is particularly effective in covering the surface of the reference model as quickly as possible. Preferably, the dedicated software guides towards the new target acquisition condition, e.g., by coloring areas of the reference model differently, or by displaying arrows indicating the desired movement of the camera, or by emitting an audible signal that becomes louder or sharper as the camera approaches the new target acquisition condition.

[0196] When the goal is reached, for example because the entire surface of the reference model is covered by the marked area, the stopping criterion is reached. A message may be sent to the user to inform them of this so that they can end the video.

[0197] The dedicated software then has enough updated images to generate an "updated" model of the user's dental arch. The final model may be generated directly from the updated images, such as a 3D scan. It may be produced by deformation of another model (e.g., a reference model), the deformation being directed so that the final model matches the updated images as closely as possible.

[0198] exist Figure 3 In the example shown, at time tf, all tooth models are shaded, indicating to the user that the last cycle of steps a) to d) has been completed and the final step of generating a final three-dimensional model from the updated images can be performed.

[0199] As is now clear, the device and method according to the invention advantageously make it possible to increase the autonomy of the user and to improve the quality and content of images acquired by users without specific knowledge in the dental field. Advantageously, they make it possible to remotely generate a 3D model of the user's dental arch. This makes it easier to determine orthodontic treatment at a distance, without requiring the user to make an appointment with a dental care professional.

[0200] Of course, the present invention is not limited to the embodiments described above and exemplified.

[0201] In particular, the invention is not limited to the use of the final model. For example, this may be used to establish a diagnosis or orthodontic treatment plan, or for manufacturing orthodontic appliances. In particular, the final model may only represent areas of interest for analyzing the user's dental condition or for determining orthodontic treatment, such as areas whose hygiene is to be assessed, the risk of recurrence in a specific area of ​​the user's dental arch, areas for which the user and / or the dental care professional wishes to plan a possible aesthetic treatment.

[0202] The reference representation is not limited to a reference model. It can be a two-dimensional image, even if a model is preferred.

[0203] In a particular embodiment of the invention, step d) of the scanning method is optional and the scanning method comprises:

[0204] - A preliminary step of determining a reference dental arch model called a "reference representation", then

[0205] - Perform multiple cycles of the following steps a) to c):

[0206] a) the user uses an image acquisition device to acquire a two-dimensional image of the dental arch under acquisition conditions, or an "updated image";

[0207] b) analyzing the updated image in order to identify a surface of the updated image representing the dental arch of the user, called the “acquired 2D surface”, then outlining and marking a surface of the reference representation corresponding to the acquired 2D surface, called the “basic marking area”, and adding this basic marking area to the “marked area of ​​the reference representation”, which brings together all these basic marking areas defined during the previous cycles, which were empty before the first cycle;

[0208] c) determining and presenting guidance information to a user for guiding the acquisition device toward a new acquisition condition suitable for a next cycle, thereby increasing the marked area of ​​the reference representation;

[0209] d) preferably, moving the acquisition device to the new acquisition condition;

[0210] After the last cycle, the method comprises a final step for generating a "final" three-dimensional model from said updated images.

[0211] Of course, the method may also comprise one or more of the above optional features that are compatible with optional step d).

Claims

1. A method for scanning a dental arch of a user, the method comprising: - a preliminary step of determining a reference dental arch model called a "reference representation", then - performing a plurality of cycles of the following steps a) to c), preferably a) to d): a) the user acquires a two-dimensional image of the dental arch, or "updated image", under acquisition conditions by means of an image acquisition device; b) analyzing the updated image in order to identify a surface of the updated image representing the dental arch of the user, referred to as the "acquired 2D surface", then outlining and marking a surface of the reference representation corresponding to the acquired 2D surface, referred to as the "basic marking area", and adding the basic marking area to the "marking area of ​​the reference representation", the marking area of ​​the reference representation bringing together all the basic marking areas defined during the previous cycles, the basic marking area being empty before the first cycle; c) determining and presenting guidance information to the user for guiding the acquisition device toward a new acquisition condition suitable for the next cycle, thereby increasing the marked area of ​​the reference representation; d) preferably, moving the acquisition device to the new acquisition condition; After the last cycle, the method comprises a final step for generating a “final” three-dimensional model from the updated images.

2. The method according to claim 1, wherein: The continuously looped updated images are extracted from a video captured by the user via a tablet computer or a cellular phone, the looping being performed in real time during the capture of the video, and at least a portion of the guidance information being displayed on a screen of the tablet computer or the phone, respectively.

3. A method according to any one of the preceding claims, wherein: The guidance information comprises the marked area superimposed on the reference representation presented on a screen of the image acquisition device, wherein unmarked areas have a different appearance than the marked areas.

4. A method according to any one of the preceding claims, wherein: The reference representation is: One or more 3D or 2D representations of a dental arch that may be photorealistic, or a "reference" model of a dental arch - indicates a group of "historical" dental arches including more than 10 historical individuals; or - by assembling a dental model in an arrangement determined by analysis of one or more analysis images of the dental arch of the user.

5. A method according to the preceding claim, wherein: In step c) of any cycle, a view of the observed reference model is presented along a viewing direction oriented relative to the reference model, the viewing direction being substantially consistent with the acquisition direction of the image updated in step a) of the cycle relative to the dental arch of the user, so that the view can be superimposed substantially in alignment with the updated image.

6. A method according to any one of the preceding claims, wherein: In any step c), the image acquisition device provides variable information according to the difference between the actual acquisition condition and the new acquisition condition.

7. A method according to any one of the preceding claims, wherein: For at least one cycle of step a), the lips and / or cheeks of the dental arch of the user are moved aside in order to expose at least one tooth to the image acquisition device.

8. A method according to any one of the preceding claims, wherein: During the cycle, the following were collected: - at least one updated image taken facing the user, at least one updated image taken on the right side of the user, and at least one updated image taken on the left side of the user; and / or at least one updated image taken facing the user, at least one updated image taken from above the user, and at least one updated image taken from below the user; and / or - at least one updated image taken when the mouth is open and at least one updated image taken when said mouth is closed.

9. A method according to any one of the preceding claims, wherein: The user is more than 100m away from any dental care professional.

10. A method according to any one of the preceding claims, wherein: The loop is stopped when more than 50% of the surface of the reference representation is represented on at least one updated image.

11. A method according to any one of the preceding claims, wherein: The final model is compared with another model representing the dental arch of the user and / or the final model is compared with the reference model, preferably by presenting the comparison to the user, for example by superimposing the final model and the reference model, preferably in an aligned manner.

12. A method according to any one of the preceding claims, wherein: The guidance information includes sound information, visual information and / or sensitive information.

13. A method according to any one of the preceding claims, wherein: The reference representation is a model, and the surface of the reference representation corresponding to the acquired 2D surface is determined by: - by searching for suitable virtual acquisition conditions for acquiring a view of said reference representation, called "reference image", and which best matches said image updated under said virtual acquisition conditions; Then - by projecting said acquired 2D surface onto said reference representation and along said viewing direction of said reference representation under said virtual acquisition conditions.

14. A method according to the preceding claim, wherein: The virtual acquisition conditions are determined by seeking to superimpose salient points of the dental arch represented on the updated image with corresponding salient points of the reference representation.

15. An apparatus for implementing the method according to any one of the preceding claims, the apparatus comprising: - Optionally, a first computer comprising a first computer program comprising program code instructions for generating said reference representation; - an image acquisition device for performing step a); - preferably, a screen for displaying guidance information during step c); - a second computer, identical to or different from said first computer, said second computer comprising a second computer program, said second computer program comprising program code instructions for performing steps b) and c); - a third computer, identical or different to said first computer and / or said second computer, said third computer comprising a third computer program, said third computer program comprising program code instructions for implementing said final step of generating said final model of said dental arch of a user from said updated image; - Preferably, a fourth computer, identical or different to said first computer and / or said second computer and / or said third computer, said fourth computer comprising a fourth computer program, said fourth computer program comprising program code instructions for comparing said final model with another model representing said dental arch of a user.

Images