Automatically apply physician preference workflows using statistical preference analysis

Abstract

Methods and apparatus for automated treatment planning, including recommender systems, quality assurance, error prevention, text mining, text matching, and treatment plan optimization.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. Provisional Patent Application No. 62 / 821,858, filed on March 21, 2019, entitled “AUTOMATIC APPLICATION OF DOCTOR'S PREFERENCES WORKFLOW USING STATISTICAL PREFERENCE ANALYSIS,” the entire contents of which are incorporated herein by reference.

[0003] Join by reference

[0004] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. Technical Field

[0005] Methods and apparatus for orthodontic treatment planning. Background Art

[0006] Orthodontic and dental treatment using a series of patient-removable appliances (e.g., orthodontic appliances, palatal expanders, etc.) is very useful for treating patients, especially for treating malocclusions. Typically, treatment planning is performed in conjunction with a dental professional (e.g., a dentist, orthodontist, dental technician, etc., referred to herein as a "user") by generating a model of the patient's teeth in the final configuration, and then dividing the treatment plan into multiple intermediate stages (steps) corresponding to the various appliances worn in sequence. This process can be interactive, and the staging, and sometimes the final target position, can be adjusted based on the limitations of tooth movement and the preferences of the dental professional. Once the final treatment plan is finalized, a series of appliances can be manufactured corresponding to the treatment plan.

[0007] The treatment planning process may include many manual steps that are complex and may require a high level of knowledge of orthodontic specifications. In addition, because the steps are performed sequentially, the process may require a significant amount of time. The manual steps may include: preparing models for digital planning, reviewing and modifying the proposed treatment plan (including staging), and appliance feature placement (which includes features placed on the teeth or on the appliances themselves). These steps may be performed before providing an initial treatment plan to a dental professional, who may then further modify the plan and send it back for additional processing to adjust the treatment plan, repeating (iterating) this process until a final treatment plan is completed, which is then provided to the patient.

[0008] Existing systems and methods for treatment planning are time consuming and may provide only limited options and control to the dental professional. Summary of the invention

[0009] The systems, methods, and / or computer-readable media described herein provide technical solutions to the highly technical problems of orthodontic treatment planning. Generally, these methods may include creating and modifying (e.g., updating) a collection of user (e.g., doctor, dentist, orthodontist, etc.) or user group preferences, and using these user or user group preferences to automatically create and / or modify orthodontic treatment plans.

[0010] Many dentists have strong clinical preferences, which should apply to most cases. These preferences can be defined once by the practitioner on their personal account, or communicated repeatedly through instructions specified for each patient. For example, "remove attachments at the end of treatment" is a very common instruction used by a large number of practitioners. This type of text instruction has an impact on case processing time because technicians must spend time reading, implementing, and rechecking known and common instructions.

[0011] Methods for creating and / or modifying a treatment plan by automatically detecting user preferences (e.g., clinical preferences) and incorporating those preferences into a treatment plan for user approval are described herein; the treatment plan can be finalized and used to produce one or a series of orthodontic appliances to treat a patient. A treatment planning engine that can perform any of these methods is also described herein.

[0012] The treatment planning engine may use a set of user or user group preferences and information about the patient's mouth (e.g., a scan of the patient's teeth) to automatically generate one or more treatment plans specific to the patient. The set of user or user group preferences may be a data structure (e.g., a data store, a database, etc.) that is accessed and / or modified by one or more treatment planning engines. The treatment planning engine may include or have access to a preference engine that may create, modify, update, and / or distribute the set of user or user group preferences.

[0013] Because the treatment plan is generated by the treatment planning engine using user or user group specific information, the treatment planning engine can automatically generate a treatment plan customized for the patient that is also customized for or for the user. The generated treatment plan can be reviewed and approved by the user; the treatment planning engine and / or the preference engine can use any modifications to the automatically generated treatment plan to update and / or modify the set of user or user group preferences.

[0014] As used herein, an orthodontic treatment plan (or simply "treatment plan") may refer to a series of steps, equipment, and / or schedule for changing a subject's dental arch to achieve or approach a desired outcome. A treatment plan may identify or describe one or more dental appliances (including dental appliances, such as but not limited to braces) that may be used to change a subject's dental arch. A treatment plan may also include or alternatively include steps for changing a subject's dental arch using and / or not using one or more dental appliances. A treatment plan may occur sequentially, for example, indicating a plurality of treatment steps, some or all of which may include orthodontic treatment appliances, such as dental appliances. In some variations, a treatment plan may include preparing a subject's dental arch (e.g., by extracting, shaping, trimming, or otherwise changing one or more teeth of the subject). A treatment plan may indicate movement (and / or non-movement) of one or more teeth of a patient, including indicating the timing or sequence (start, duration, completion) of the movement. A treatment plan may include steps for designing and / or manufacturing one or more dental appliances (including an ordered series of dental appliances). A treatment plan may include a dental appliance schedule indicating when to wear one or more dental appliances.

[0015] Any method or apparatus (including software, hardware, and firmware, such as a treatment planning engine) can receive a collection of patient-specific information, such as a description (e.g., a model, 3D scan, mold, etc.) of a patient's dental arch (including all or part of the patient's upper and / or lower dental arches) and instructions from a user associated with the patient (e.g., an orthodontist, a dentist, a physician, a dental technician, etc.). In some variations, a description of user instructions for the patient can be provided. The user instructions can be a description of the user's general and / or specific preferences for one or a class of dental treatments, or can be specific to the patient. The user instructions can include, for example, tooth movement restrictions (e.g., indicating which teeth should not be moved as part of the treatment), whether interproximal removal (IPR) should be used and / or how to perform IPR, when to perform IPR during treatment or where to perform IPR, whether attachments should be used, and if so, where the attachments should be placed (e.g., on which teeth), changing the spacing between teeth, extraction instructions (including which teeth to extract and at what stage during the treatment to extract them), leveling strategies (e.g., "incisal edge alignment" or "gingival edge alignment"), etc. The user instructions may indicate any suitable number of preferences / instructions (including one or more). Instructions may be categorical and / or conditional (eg, instructions that are dependent on one or more other conditions).

[0016] A treatment planning engine may refer to software, hardware, and / or firmware (or some combination of these) that receives treatment planning instructions and / or patient information (e.g., a digital model of a patient's teeth), and may apply the treatment planning instructions to the patient information to generate one or more treatment plans. Any suitable digital model of a patient's teeth may be used, including a 3D volume scan, such as from an intraoral scanner.

[0017] Methods for generating a treatment plan for orthodontic treatment are described herein. Any of these methods can be computer-implemented methods. For example, described herein is a computer-implemented method comprising: creating a user preference database by: identifying a reference key preference text statement corresponding to a clinical behavior for treating a patient's teeth, accessing a user-specific text instruction set including a plurality of dental treatment instructions, each of the plurality of dental treatment instructions being associated with a specific user, identifying a user-specific text key preference statement from a user-specific text instruction set by matching the reference key preference text statement with a user-specific text instruction from the user-specific text instruction set within a specified probability range, and adding an entry to the user preference database that links the user-specific text key preference statement, the specific user associated with the user-specific text key preference statement, and the clinical behavior for treating the patient's teeth; receiving a description of the patient's dentition; receiving a patient treatment instruction for performing an orthodontic procedure on the patient from a dental professional; and automatically generating a dental treatment plan using the user preference database.

[0018] For example, a computer-implemented method may include: creating a user preference database by: identifying a reference key preference text statement corresponding to a clinical behavior for treating a patient's teeth, accessing a user-specific text instruction set including a plurality of dental treatment instructions, each of the plurality of dental treatment instructions being associated with a specific user, identifying a user-specific text key preference statement from a user-specific text instruction set by matching the reference key preference text statement with the user-specific text instructions from the user-specific text instruction set within a specified probability range using both a machine learning agent and a similarity retrieval, and adding an entry to the user preference database linking the user-specific text key preference statement, the specific user associated with the user-specific text key preference statement, and the clinical behavior for treating the patient's teeth; receiving a description of the patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0019] A computer-implemented method may include: creating a user preference database, wherein the database includes multiple user identifiers, one or more user-specific text key preferences corresponding to each user identifier, and clinical behaviors corresponding to each user-specific text key preference; receiving a description of a patient's dentition; receiving user-specific text instructions for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan by interpreting the user text instructions using the user preference database to match one or more user-specific text key preferences with one or more phrases in the user-specific text instructions and incorporating the clinical behaviors corresponding to each matched user-specific text instruction into the dental treatment plan.

[0020] In any of these methods, automatically generating a dental treatment plan using a user preference database may include: finding any user-specific text key preference statements from a user preference database associated with a dental professional; and incorporating into the dental treatment plan clinical actions for treating the patient's teeth that correspond to any found user-specific key preferences, any found user-specific key preferences matching phrases in the patient's treatment instructions.

[0021] Any suitable method for identifying user-specific text key preference statements from a user-specific set of text instructions may be used, including: using a machine learning agent to match a reference key preference text statement with a user-specific text instruction from a user-specific set of text instructions. For example, any of these methods may include: training a machine learning agent using a set of positive cases selected from a plurality of dental treatment instructions and a set of negative cases selected from a plurality of dental treatment instructions. The machine learning agent may be implementing a bag-of-words variation suitable for analyzing sentences. Using a machine learning agent to identify user-specific text key preference statements to match a reference key preference text statement with a user-specific text instruction from a user-specific set of text instructions may also include: using similarity retrieval simultaneously with the machine learning agent. For example, similarity retrieval may include a Damerau-Levenshtein distance metric. In any of these methods, identifying a user-specific text key preference statement from a user-specific set of text instructions may include: using a similarity retrieval agent to match a reference key preference text statement with a user-specific text instruction from a user-specific set of text instructions.

[0022] Any of these methods may include collecting a user-specific set of text instructions from a set of historical treatment data.Receiving a description of the patient's dentition may include receiving a digital model of the patient's dentition.

[0023] The present invention also describes a system that can perform any of the methods described herein. For example, the system may include: one or more processors; a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method, including: creating a user preference database by: identifying a reference key preference text statement corresponding to a clinical behavior for treating a patient's teeth, accessing a user-specific text instruction set including a plurality of dental treatment instructions, each of the plurality of dental treatment instructions being associated with a specific user, identifying a user-specific text key preference statement from a user-specific text instruction set by matching the reference key preference text statement with the user-specific text instruction from the user-specific text instruction set within a specified probability range, and adding an entry to the user preference database that links the user-specific text key preference statement, the specific user associated with the user-specific text key preference statement, and the clinical behavior for treating the patient's teeth; receiving a description of the patient's dentition; receiving a patient treatment instruction from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0024] A system may include: one or more processors; a memory coupled to the one or more processors, the memory configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method, including: creating a user preference database by: identifying a reference key preference text statement corresponding to a clinical behavior for treating a patient's teeth, accessing a user-specific text instruction set including a plurality of dental treatment instructions, each of the plurality of dental treatment instructions being associated with a specific user, identifying a user-specific text key preference statement from a user-specific text instruction set by matching the reference key preference text statement with the user-specific text instructions from the user-specific text instruction set within a specified probability range using both a machine learning agent and similarity retrieval, and adding an entry to the user preference database that links the user-specific text key preference statement, the specific user associated with the user-specific text key preference statement, and the clinical behavior for treating the patient's teeth; receiving a description of the patient's dentition; receiving a patient treatment instruction from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0025] In some embodiments, a system may include: one or more processors; a memory coupled to the one or more processors, the memory configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: creating a user preference database, wherein the database includes multiple user identifiers, one or more user-specific text key preferences corresponding to each user identifier, and a clinical behavior corresponding to each user-specific text key preference; receiving a description of a patient's dentition; receiving user-specific text instructions for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan by: interpreting the user text instructions using the user preference database to match one or more user-specific text key preferences with one or more phrases in the user-specific text instructions; and incorporating the clinical behavior corresponding to each matched user-specific text instruction into the dental treatment plan.

[0026] Also described herein are methods and apparatus (e.g., systems) for using statistical preference analysis. For example, a computer-implemented method may include: creating a user preference database by: performing statistical preference analysis to identify key preference text statements associated with a single user and corresponding clinical behaviors for treating a patient's teeth from a user-specific set of text instructions including a plurality of dental treatment instructions, each of which is associated with a specific user, adding an entry to the user preference database that links the key preference text statement, the single user associated with the key preference text statement, and the corresponding clinical behaviors for treating the patient's teeth; receiving a description of a patient's dentition; receiving a patient treatment instruction from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0027] Performing a statistical preference analysis can include confirming that the clinical behavior used to treat a patient's teeth remains approximately constant during a time period (e.g., constant during 70% of the time period, during 75% of the time period, during 80% of the time period, during 85% of the time period, during 90% of the time period, during 95% of the time period, etc.), where the time period can be the duration of a cluster of recent treatment plans.

[0028] For example, a computer-implemented method may include: creating a user preference database by: collecting user-specific text instructions including multiple dental treatment instructions from a historical treatment data set, each of which is associated with a clinical dental procedure associated with a single user; identifying statistically consistent clinical behaviors for treating a patient's teeth and corresponding key preference text statements during a period of time from the clinical dental procedure instructions; adding the following entry to the user preference database, which links the key preference text statements, the single user, and the corresponding clinical behaviors for treating the patient's teeth; receiving a description of the patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0029] A computer-implemented method may include: creating a user preference database by: collecting user-specific text instructions including multiple dental treatment instructions from a historical treatment data set, each of which is associated with a clinical dental procedure associated with a single user; identifying statistically consistent clinical behaviors for treating patient teeth and corresponding key preference text statements during a time period from the clinical dental procedure instructions, wherein the time period includes the duration of a recent cluster of clinical behaviors for treating patient teeth; adding the following entry to the user preference database, which links the key preference text statements, the single user, and the corresponding clinical behaviors for treating the patient's teeth; receiving a description of the patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0030] Automatically generating a dental treatment plan using a user preference database may include: finding any user-specific text key preference statements from a user preference database associated with a dental professional; and incorporating clinical actions for treating the patient's teeth into the dental treatment plan, the clinical actions corresponding to any found user-specific key preferences, any found user-specific key preferences matching phrases in the patient's treatment instructions.

[0031] The statistical preference analysis may include identifying clinical behaviors for treating teeth of a patient and corresponding key preference text statements from clinical dental procedure instructions, wherein the clinical behaviors for treating teeth of a patient are statistically consistent during a period of time.

[0032] Any of these methods may include collecting, from a historical treatment data set, user-specific textual instructions comprising a plurality of dental treatment instructions, wherein each dental treatment instruction is associated with a clinical dental procedure associated with a single user.

[0033] Performing the statistical preference analysis may include performing the statistical preference analysis after confirming that the clinical behavior for treating the patient's teeth is statistically consistent during a recent behavior detection period and after limiting the statistical preference analysis to a time period.

[0034] Any of these methods may include manually validating the key preference text statements and corresponding clinical actions for treating the patient's teeth before adding them to the user preference database.Receiving a description of the patient's dentition may include receiving a digital model of the patient's dentition.

[0035] The present invention also describes a system configured to perform any of the methods described herein. For example, the system may include: one or more processors; a memory coupled to the one or more processors, the memory configured to store computer program instructions, the computer program instructions when executed by the one or more processors performing a computer-implemented method, including: creating a user preference database by: performing statistical preference analysis to identify key preference text statements associated with a single user and corresponding clinical behaviors for treating a patient's teeth from a user-specific text instruction set including a plurality of dental treatment instructions, each of the plurality of dental treatment instructions being associated with a specific user, adding an entry to the user preference database that links the key preference text statements, the single user associated with the key preference text statements, and the corresponding clinical behaviors for treating the patient's teeth; receiving a description of a patient's dentition; receiving a patient treatment instruction from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0036] A system may include: one or more processors; a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method, including: creating a user preference database by: collecting user-specific text instructions including a plurality of dental treatment instructions from a historical treatment data set, the plurality of dental treatment instructions each being associated with a clinical dental procedure associated with a single user; identifying statistically consistent clinical behaviors for treating a patient's teeth and corresponding key preference text statements during a time period from the clinical dental procedure instructions; adding an entry to the user preference database that links the key preference text statements, the single user, and the corresponding clinical behaviors for treating the patient's teeth; receiving a description of a patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0037] Any of these systems may include: one or more processors; a memory coupled to the one or more processors, the memory configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method, including: creating a user preference database by: collecting user-specific text instructions including a plurality of dental treatment instructions from a historical treatment data set, the plurality of dental treatment instructions each being associated with a clinical dental procedure associated with a single user; identifying statistically consistent clinical behaviors for treating a patient's teeth and corresponding key preference text statements during a time period from the clinical dental procedure instructions, wherein the time period includes the duration of a recent cluster of clinical behaviors for treating the patient's teeth; adding an entry to the user preference database that links the key preference text statements, a single user, and the corresponding clinical behaviors for treating the patient's teeth; receiving a description of a patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The novel features of the invention are particularly set forth in the appended claims.A better understanding of the features and advantages of the invention may be obtained by referring to the accompanying drawings and the following detailed description which sets forth illustrative embodiments in which the principles of the invention are utilized.

[0039] Figure 1 A workflow of a first method for determining a set of preferences for a user or user group is shown.

[0040] Figure 2 A method of automatically generating a treatment plan and creating a series of dental appliances for treating a patient is shown.

[0041] Figure 3 is a graph illustrating improvements in automatic treatment plan generation (illustrating reduced user modifications required for automatically generated treatment plans) using a variation of a method for automatically generating treatment plans as described herein.

[0042] Figure 4 are examples of workflows (eg, methods) implementing monitoring a set of user or user group preferences and / or automatically generating a treatment plan using a set of user or user group preferences as described herein.

[0043] Figure 5 is a report showing the statics of an automatically generated treatment plan as described herein.

[0044] Figure 6 is another example of a report showing details of the statics of a treatment plan automatically generated as described herein.

[0045] Figure 7 is another example of a workflow of the first method of determining a set of preferences for a user or user group.

[0046] Figure 8 is a table showing instructions from different users (“preference texts”) that the machine learning agent has classified as either a positive or negative class.

[0047] Fig. 9 is a diagram illustrating an example of an analysis of key preference terms used by users over time.

[0048] Fig. 10A are examples of three text instructions that have been labeled with different values ​​(shown as text units or words with numerical values ​​above) using machine learning techniques for text matching between phrases of words.

[0049] Fig. 10B The probability of a match between each of the three sentences and a target sentence (eg, the key preference sentence "Start Precision cuts at the stage #") is shown.

[0050] Fig.11 is a table showing an example of the results of a similarity search between different references and the found text from the user's text description, the table showing the percentage of similarity.

[0051] Fig.12 is another example of a workflow of the first method of determining a set of preferences for a user or user group.

[0052] Fig.13 is a table showing an example of similarity search for word matching using Damerau-Levenshtein (DL) distance estimation.

[0053] Fig.14 is an example showing the use of semantic similarity to calculate similarity estimates.

[0054] Fig.15 An example of implementing a similarity retrieval technique such as DL distance is shown.

[0055] Fig.16A is another example of a workflow of the first method of determining a set of preferences for a user or user group.

[0056] Fig. 16B is another example of a workflow of the first method of determining a set of preferences for a user or user group.

[0057] Fig.17An example of a statistical preference analysis for a specified user and a specified clinical behavior is shown.

[0058] Fig.18 is another example of a statistical preference analysis for a given user and a given clinical behavior.

[0059] Fig.19 is an example of a block diagram describing a variation of the text feature qualification process.

[0060] Fig. 20 A method of creating a user preference database is shown.

[0061] Fig.21 A method for automatically generating a treatment plan using a user preference database is shown.

[0062] Fig. 22 Another method of creating a user preference database is shown.

[0063] Fig.23 A variation of an automated treatment planning system as described herein is shown. DETAILED DESCRIPTION

[0064] Many medical professionals have strong preferences for the same standard medical treatment they perform. For example, a dental medical professional (e.g., a dentist, orthodontist, dental technician, etc., referred to herein as a "user") can assist in designing a treatment plan for moving or aligning a patient's teeth. As part of the treatment plan, the user can provide patient dental information (e.g., a model or scan of the patient's dental arch) as well as instructions and / or preferences, and can receive a suggested treatment plan for aligning the patient's teeth. In many cases, the user can then modify the treatment plan, typically by communicating with the technician, so that the treatment plan can be adjusted and provided to the user again. This process can be repeated multiple times. Although it is generally desirable to simplify this process and reduce the number of times the treatment plan is corrected or adjusted (collectively referred to as "modification"), it is also desirable to ensure that the final treatment plan meets the user's preferences and treatment goals. The methods and devices described herein can help ensure that the user's explicit and implicit preferences are considered as accurately as possible in the automatically generated treatment plan.

[0065] User preferences may be explicitly defined by the user and / or interpreted based on analysis of the user's past behavior when treating a patient. The methods and apparatus described herein (e.g., devices and systems, including but not limited to software, firmware, and hardware) may construct a set of preferences specific to a user (and / or user group), may maintain and / or update the set of user and / or user group preferences, and may apply the set of user and / or user group preferences to automatically or semi-automatically generate a treatment plan that is specific to a patient and customized based on user preferences. Although the methods and apparatus described herein may be specific to a particular user (e.g., a single designated user), in some variations, the methods and apparatus described herein may be applied to a group of two or more users (e.g., a practice group), or a designated user or user group may choose to apply the preferences of another user or user group.

[0066] The set of user and / or user group preferences can be constructed as a database, data store or any other suitable data structure. Therefore, the set of users and / or user groups can be referred to as a user and / or user group preference data structure, and can be but not necessarily a list. The set of user and / or user group preferences can be stored in a single location (e.g., on a remote server) or distributed in multiple locations. The set of user and / or user group preferences can be locally copied and updated from one or more different sites.

[0067] A collection of user and / or user group preferences may include information including multiple users (or user groups) and associated instructions (which may be referred to as key preferences) and one or more associated clinical behaviors. For example, a particular physician, Dr. X, may be associated with one or more key preferences (e.g., no attachments in overcorrection, extractions between stage 2 and stage 3, etc.), and each key preference may be associated with one or more (and in some cases, a single) clinical behaviors that may be applied to a treatment plan (e.g., remove attachments in overcorrection, perform extractions at stage 3, etc.). The clinical behaviors may be standardized, for example, into a domain-specific orthodontic treatment language (e.g., see U.S. patent application Ser. No. 16 / 399,834, filed on April 30, 2019, entitled “SYSTEMS AND METHODS FOR TREATMENT USING DOMAIN-SPECIFIC TREATMENT PROTOCOLS,” the entire contents of which are incorporated herein by reference).

[0068] In general, methods and devices for creating and / or modifying (including updating) a set of user and / or user group preferences are described herein. A set of user and / or user group preferences may be initially or partially based on historical data analysis from one or more users. The methods and devices described herein may be specific to a single user or user group. However, in some variations, common or standard key preferences (e.g., instructions) from other users or user groups may be used for some or all users at least initially. Therefore, the methods and devices described herein may form a set of user and / or user group preferences, may monitor and update a set of user and / or user group preferences, and may apply a set of user and / or user group preferences to automatically generate a treatment plan.

[0069] For example, the methods and devices described herein can provide automation of treatment planning by first identifying the key preferences of a user (or user group). The process can be automatic. Identifying the key preferences of a particular user or user group is a complex process, and the methods and devices described herein provide variations (some or all of which can be combined in whole or in part) to build and modify a set of user and / or user group preferences to include all key preferences or relevant subsets of key preferences for each user or user group. Each user or user group in the set of user and / or user group preferences can include the same or different key preferences.

[0070] For example, in some variations, in addition to information about a particular patient (e.g., dental information), a user may also provide a request (e.g., a written request) for some key preferences that correspond to a certain type of action that the user wishes to include in or be considered in the treatment plan. This may be included as part of a prescription form (e.g., a "treatment RX form") and may be written in free text (using a user-specific language, jargon, or symbols). Without automation, these key preferences must be interpreted by a technician and applied to the design of a treatment plan, or across multiple treatment plans specific to that user or user group. However, users may express their key requests in very different ways (e.g., "remove all attachments starting from the last appliance of active treatment", "end attachments after the second-to-last active phase, so there are no attachments for the last two active phases", "remove all attachments at the first passive appliance", etc.). In addition, users may write their instructions in different languages ​​(e.g., English, French, Spanish, Chinese, German, Japanese, Portuguese, Turkish, Korean, etc.). For example, "remove attachments at the end of treatment" is a very common instruction used by many or all users, but may be expressed in many different ways, which may negatively impact treatment processing because technicians would have to spend time reading, implementing, and rechecking even these well-known and common instructions if performed manually. Furthermore, in some cases, even key preferences that are expressed very similarly may differ between different users.

[0071] The methods and devices described herein can address these problems in a variety of ways. For example, these methods and devices can build and maintain a collection of user and / or user group preferences by text mining user instructions from a collection (e.g., a database) of approved treatment plans associated with a user or user group and instructions specific to the approved treatment plans (which can be referred to as historical treatment data). Text mining can include text matching (including but not limited to text matching by semantic similarity) and / or machine learning specifically applicable to key preferences. In addition to text mining, these methods and devices can also simultaneously identify user-specific and / or common clinical behaviors associated with each key preference identified. The associated clinical behavior can be based on verified clinical behaviors from approved treatment plans.

[0072] Alternatively or additionally, the set of user and / or user group preferences may include one or more key preferences determined from an analysis of a user's historical treatment plans that were not explicitly expressed by the user. For example, in some variations, the method and / or apparatus may include a statistical analysis of clinical behavior from a user's treatment plans; in some variations, the statistical analysis may be based on current (e.g., most recent or consistent with the analysis determination) user practice from the user's treatment plans. The method or apparatus may generate key preferences and associate them with identified clinical behaviors.

[0073] Thus, the methods and apparatus described herein can automatically and / or semi-automatically establish and maintain a set of user and / or user group preferences from a historical treatment data set. The historical treatment data set can be a centrally organized (curate) database (e.g., storing approved treatment plans associated with a user or user group and any instructions from the user). In the case where there are one or more associated treatment plans associated with a particular user in the historical treatment data, in some variations, the method or apparatus can first identify text instructions and identify one or more key preferences. The retrieval of one or more key preferences specific to a particular user (e.g., a specified user) can be guided by key preferences identified from written instructions and / or treatment plans of other users. For example, common key preferences can be determined for other user groups; in some variations, these other users can be part of a user group, or they can be related or unrelated to the specified user.

[0074] The historical treatment data for the specified user may then be analyzed as described herein to identify key preferences and corresponding clinical behaviors from the historical treatment data; this information may be input into a set of user and / or user group preferences. As described above, and described in more detail below, in some variations, instead of or in addition to identifying key preferences from text instructions for the specified user in the historical treatment data, the method or apparatus may review the behavior of the specified user (e.g., clinical behaviors from an approved treatment plan) and associate with, for example, common key preferences. Thus, the methods and apparatus described herein may add to or modify a set of user and / or user group preferences based on text or user behavior.

[0075] For example, described herein are methods and apparatus for forming a set of user and / or user group preferences that includes a list of users (e.g., doctors, dentists, orthodontists, etc.) and validated text instructions (key preferences) associated with one or more clinical actions. Figure 1 In this example, the method or apparatus may first find users for whom certain clinical behavior patterns can be determined and applied. Figure 1As shown, the set of user and / or user group preferences is called "doctor list with verified text instructions associated with clinical behavior" 109 in this example. In this example, the list (the set of user and / or user group preferences) is formed by analyzing the text instructions of the specified user 101. By finding and including variations of words and terms in the text instructions corresponding to the key preferences, these instructions become the core of the set of user and / or user group preferences. In this example, starting with the specified user or user group 101, the method or device can then identify and collect text instructions 103, for example, from a database of historical treatment data. The set of user and / or user group preferences can be iteratively expanded, for example, by finding new text instructions that reference the specified key preferences, forming a list of the most likely candidates based on the text instructions, and ranking them (for example, as described in detail below, by one or more methods, including machine learning techniques and / or text matching, etc.). Ranking can include automatically excluding those that are below a threshold ranking or likelihood probability as key preferences.

[0076] The process of retrieving key preferences from text instructions 103 can, for example, start from the most common and / or necessary key preference words or phrases based on the collected text instructions. As described above, initially, groups of various different users can be checked to identify common or popular key preferences to form instruction cores. For example, users whose text instructions and comments use specific keywords corresponding to key preferences can be identified from the collected text instructions (e.g., from historical treatment data or a subset thereof). Therefore, a list 105 of instruction variants (possible key preferences) can be formed. Examples of different ways of collecting possible key preferences (e.g., identifying possible key preferences) and rating / ranking these possible key preferences or otherwise distinguishing between these possible key preferences are described in more detail below. Based on the results from the previous step, a list of possible key preferences can be generated. The process can include analysis of the doctor's treatment cases (e.g., the number of such cases), text instructions, and other indicators.

[0077] The method or apparatus may then optionally associate the key preferences with clinical behaviors from historical treatment data. For example, the apparatus or method may perform manual instruction tagging 107 from the list of possible key preferences formed. Text tagging (including manual text tagging) may be performed based on the key preferences. In some variations, the text tagging may refer to the association of the key preferences with specific clinical behaviors. The process may be semi-automatic (e.g., although automatically generated, it is assisted or reviewed by a technician). Figure 1As shown, the process may optionally include a verification step 107'. Verification may include manual review (e.g., by a technician). For example, the key preferences (e.g., instructions) marked may be verified by a responsible person with the required level of clinical knowledge.

[0078] Once a set 109 of user or user group preferences has been constructed (in this example, a list of doctors with validated text instructions associated with clinical behaviors), the set of user or user group preferences can be used to detect key preferences and apply corresponding clinical behaviors to generate a treatment plan for the user. Figure 2 An illustration of a method for automatically generating a treatment plan customized for a patient for a specified user using a set of user or user group preferences is shown. The set of user or user group preferences (e.g., "a list of doctors with verified text instructions associated with clinical behaviors") is a key-value structure for each user, where the value is the clinical behavior and the key is the text instruction corresponding to the behavior. In some variations, a text instruction matching process (the same process that can be used to identify key preferences when creating a set of user or user group preferences, or a different text matching process) can compare the user's text instructions for treating a patient with the set of user or user group preferences. When a sufficiently close match between the user's treatment text is found for the key preferences associated with the specified user or user group, the corresponding clinical behavior can be automatically included in the new treatment plan. Therefore, in some variations, the clinical behavior is applied only when the treatment text matches the key preference. Typically, text matching can be case-insensitive and punctuation can be ignored.

[0079] For example, Figure 2 As shown, a method for automatically generating a treatment plan is shown. In this example, it is assumed that a set of user or user group preferences has been generated 205; alternatively or additionally, the method may include generating a set of user or user group preferences, such as Figure 1 As shown schematically in Figure 2In the present invention, a device (e.g., a treatment planning engine) may receive patient-specific information (e.g., a description of the patient's dental arch, including one or more scans, etc.) and treatment text from a user, including a request for a new treatment plan for the patient 201, in this example, the user is "Doctor N". The method or device may then determine the user's (Doctor N's) treatment preferences 203, which may include accessing a set of user or user group preferences and / or generating a set of user or user group preferences, and identifying the user Doctor N from the included users. In some variations, if the specified user (Doctor N in this example) is not already part of the set of user or user group preferences 205, then the user may be added to the set of user or user group preferences, as described above. For example, the method or system may examine a set of historical treatment data to identify key preferences and associated clinical behaviors, and add them to the set of user or user group preferences. In some variations, when a specified user requests a new treatment plan (or a treatment plan for a new case), the system may update the set of user or user group preferences for the specified user.

[0080] The user preferences included in the set of user or user group preferences 205 can then be compared 209 to the treatment text 207 by comparing the treatment text to the key preferences in the set of user or user group preferences ("analyzed text") . This can include text matching and / or machine learning, as will be described in more detail below. In the event that a sufficient match is identified, the corresponding clinical behavior from the set of user or user group preferences can be automatically included in the treatment plan 211. Once all key preferences have been identified (and / or updated and identified), the treatment plan can be automatically generated and sent to the treatment building 213 or to the user for review before being sent to the treatment building. If the user makes modifications to the treatment plan, these modifications (as new or updated treatment text) can be added and automatically reviewed (e.g., by analyzing the text 209) as described above.

[0081] As briefly mentioned above, in any of the variations described herein, a set of user or user group preferences can be updated and / or created by reviewing treatment text to identify key preferences. For example, in variations where the historical treatment data set includes a sufficient number (and / or sufficiently recent) of cases (which can be statistically determined as described below), the method or apparatus can compare the text of the treatment text with the user's previous text instructions from the historical treatment data set to identify any user-specific key preferences. When there is a match of sufficient confidence (which can be manually or semi-manually reviewed or organized), the key preferences of the user or user group can be added to the set of user or user group preferences.

[0082] In any of the methods and apparatus described herein, a user may also or alternatively select a user group from which key preferences may be selected. For example, a designated user may select (e.g., when requesting a new or updated treatment plan) other users or user groups (e.g., key opinion leaders and / or practice groups) that they follow. In some variations, the method or apparatus may then apply key preferences from the other users or user groups, and / or may identify key preferences for the designated user that correspond to those key preferences identified for the other users or user groups, and apply corresponding clinical behaviors to each of those key preferences described by the other users or user groups from the set of user or user group preferences.

[0083] In practice, the above method can ensure that the application of key preferences is based on accurate text that matches pre-validated text patterns. This can be used to accurately and efficiently generate one or more treatment plans. In addition, the duration of the treatment planning process can be significantly reduced. For example, Figure 3 An example of a graph is shown below, which shows that the percentage of cases (treatment plans) that required changes to the treatment plan based on user modifications to the treatment plan (which may require human attention) was significantly reduced after implementing the above method, from 12.48% to 8.5%.

[0084] monitor

[0085] In any of the methods and devices described herein, the identified key preferences and / or clinical behaviors can be checked and verified, for example, manually or semi-automatically. This can reduce the risk of incorrect detection and errors that may prevent users from accepting automatically determined treatment plans. In addition, users may change their clinical behaviors so that previously detected key preferences may become invalid. Therefore, described herein are monitoring (quality monitoring) methods and tools for performing monitoring. These methods and / or tools can be implemented as the methods and devices described herein, and can help detect situations where the user's text instructions and / or clinical behaviors (e.g., applied treatment options) are different from the treatment plan that is ultimately accepted; for example, different from the user's intentions. In general, these monitoring techniques or tools can aggregate data from different treatment plan creation stages, and can provide reports or indicators that indicate the extent to which clinical behaviors correspond (or do not correspond) to the user's text instructions.

[0086] Monitoring can be used as a quality control system and can assist in the clinical preference detection workflow. For example, as part of a quality control system, it can provide factual information about the robustness of an automatic preference detection technique (and / or a method or apparatus implementing an automatic preference detection technique). If a preference detection and / or clinical behavior associated therewith (e.g., in a set of user or user group preferences) is incorrect, such situation can be further processed, and / or the set of user or user group preferences can be updated or corrected.

[0087] For example, Figure 4 Examples of workflows (e.g., methods) for implementing monitoring of a set of user or user group preferences and / or automatically generating a treatment plan using a set of user or user group preferences as described herein are shown. Figure 4 In addition to the recent behavior detection algorithm, as part of the clinical preference detection workflow, a monitoring tool is also included and can detect how and when the user's behavior changes. In this exemplary method, a treatment plan (multiple treatment plans) can be automatically or semi-automatically generated, for example, as described above with reference to Figure 1-3 These methods may also include creating 403 and / or maintaining a set 405 of user or user group preferences (referred to in this example as "a list of doctors with validated text instructions associated with clinical behavior"). Figure 2 As described in , when provided with a treatment planning engine 407 that receives, for example, patient-specific information of a patient's dental arch and user instructions / requests (e.g., treatment text), a set of user or user group preferences can be used to automatically generate one or more treatment plans. Thus, the treatment planning engine can access the set of user or user group preferences, and by applying one or more techniques for text matching and / or machine learning, key preferences and associated clinical behaviors can be identified and applied to automatically generate a treatment plan. The treatment plan can then be provided to the user for verification / finalization, and once finalized, it can be added to a clinical database 409 (e.g., which can include, but is not limited to, a historical treatment data set). The clinical database and / or the treatment planning engine can also include one or more indicators that indicate how many iterations are required before the treatment plan is finalized and accepted, and / or what corrections or requests the user made before the treatment plan is finalized and / or accepted. Then, the monitoring tool 411 can access the clinical database and / or the set of user or user group preferences to review the treatment plan based on data from the clinical database and analyze and provide a report 413 including corrective measures, which review data can be used to improve and / or correct the set of user or user group preferences.

[0088] For example, as a result of the automatic preference application workflow, a set of user or user group preferences may be generated and / or modified, and the set of user or user group preferences may provide information about whether a particular treatment option should be automatically applied to the current case. During the treatment plan creation process, the clinical database may collect statistical information, including but not limited to data about all treatment options that were applied or not applied. The monitoring tool may use this data to provide a report on how effective the automatic preference application was over a specified period of time, for example, how effective the textual matching was when comparing the set of user or user group preferences to the user's instructions, and / or how the set of user or user group preferences was completed or corrected. For example, a report from the monitoring tool may include any one or more of the following: (1) the total number of cases initially submitted; (2) the number of initial cases with recognized textual instructions and corresponding treatment options that were automatically applied; (3) the number of accepted cases that retained the automatically applied treatment options; (4) the number of accepted cases that changed the automatically applied treatment options for some reason; (5) the number of accepted cases that did not have automatically applied treatment options, where these options were manually applied during treatment plan creation; (6) and the number of accepted cases that did not have automatically applied treatment options, where these options were not manually applied. The report may also include or indicate the user corresponding to any case that was modified.

[0089] This data can be generated generally for all users or for each specific user. The data from items (1) and (2) can be used to evaluate the impact of the automatic preference application workflow on treatment planning. For example, if the text instructions are changed or the user's clinical behavior is changed (e.g., the user no longer uses the corresponding feature).

[0090] Data from items (3) to (6) can provide good / bad detection rates. Cases from items (3) and (6) may follow a positive scenario: if there is a text instruction, the treatment option is expected to be applied, and if there is no such text instruction, the treatment option will not be applied. The case from item (4) may follow a negative scenario: the detected preference does not fit the user's actual needs. This can be investigated in detail and the text instruction may be excluded from the list. The case from item (5) can be labeled neutral; there are many cases when the treatment option applied is case-specific and the user has not stated this in the text. However, if the user has many such cases, this may indicate that the system may be flagging the user as knowing such anomalies.

[0091] The monitoring tools described herein can provide different types of reports, including reports to the user and / or feedback to the treatment planning engine. Figure 5) shows an example of a brief report with statistics for multiple users. In some variations, a more detailed report with statistics for each user can be generated, an example of such a report is shown in Table 2 ( Figure 6 ). Positive cases are approved and negative cases represent cases that are not approved (and can be converted to positive cases through correction). For example, if a user has a relatively large negative case ratio (or negative case percentage), the user or an administrator (e.g., operating a treatment planning engine) can find out the reason for the user's cases and determine whether the key preferences should be excluded and / or whether the associated clinical behaviors should be modified and / or excluded. In some variations, a low number of positive cases (e.g., positive case percentage) and a high neutral case ratio (e.g., neutral case percentage) may indicate that the user's behavior has changed and all or some of the key preferences are outdated and should be updated or removed.

[0092] Text Retrieval

[0093] The methods and devices described herein can identify key preferences from user text instructions (e.g., treatment text instructions) based on matching treatment text instructions with known text patterns corresponding to key preferences and / or clinical behaviors. Commonly referred to as text retrieval or text matching, described herein are various techniques that can be applied to identify key preferences during one or more of the following: creating an initial set of user or user group preferences for one or more users or user groups by browsing historical treatment data sets and comparing the historical text instructions of a specified user with known key preferences or comparing the historical text instructions of a specified user with other historical text instructions of a specified user; and / or comparing the current text instructions of a specified user with key preferences already in the set of user or user group preferences. Therefore, text retrieval (which may also be referred to herein and may include text matching) may be suitable for use in any of the following methods and devices, in which user text is compared with other texts to identify matches between parts of text (e.g., key preference text and / or clinical behavior text) and / or score them. The text retrieval / text matching techniques described herein are particularly suitable for extended text patterns and can provide confidence levels that can be used to indicate positive or negative matches.

[0094] In some variations, the techniques for text matching described herein may include one or more machine learning techniques suitable for use. For example, described herein are machine learning techniques for finding a match (or for scoring a match) between a user's text instructions and one or more key preferences. In general, one or more text matching techniques suitable for matching identified or inferred key preference text to user text may be incorporated (e.g., incorporated into a workflow) into any of the devices and methods described herein. For example, machine learning techniques for matching text to identified or inferred key preferences may be integrated into existing workflows, such as Figure 7 As shown schematically in FIG.

[0095] Figure 7 Similar to Figure 1 , but includes the use of machine learning 703 as part of the collection 103 and retrieval 105 of text instructions. In this example, a set of user or user group preferences is created by using machine learning text matching techniques and identifying key preferences in user text instructions, and key preferences from a set of user or user group preferences can be completed by machine learning techniques. For example, in some variations, the text matching engine can be configured as a machine learning agent (e.g., an engine) that is trained based on key preferences identified from a historical treatment data set for each clinical behavior. In addition, the trained model can analyze the user's text instructions in historical treatment data extending from the most recent time period to a specified time period determined to be a time period that remains accurate (e.g., as determined by statistical analysis as described below (e.g., indicating consistent users with the same key preferences and / or clinical behaviors)). The result can be a list of text instructions (presumed key preferences) with a high probability corresponding to the target clinical preference. A technician or other person with sufficient clinical knowledge can review (e.g., organize) the identified putative key preferences; thus, after validating each identified putative key preference and / or clinical behavior, the putative clinical behavior can be manually labeled (at least initially, when training the engine). The resulting validated key preferences can be saved as a final list of key preferences associated with clinical behaviors, and these instructions can be used to expand the training and test data sets to further machine learning of key preferences.

[0096] For example, a machine learning (ML) engine may identify, for example, a collection of text instructions collected at an example of a key preference (e.g., extracting teeth at a defined treatment stage). For purposes of ML, these text instructions may be divided into two data sets: instructions that express a user's desire to perform an extraction at a specific treatment stage (e.g., a positive data set), and any text instructions that are not associated with that preference type (e.g., a negative data set). Examples of text instructions that may be considered part of a positive data set may include, for example: "Extract Ext at stage 2," "Extraction will be performed after the appliance is placed after tray 1," "Aligners, extract from 2," and "Delay extraction until second appliance." Text instructions that may be part of a negative data set may include, for example: "No extraction," "Extract 4s at a young age," and "Trim all appliances buccal and lingual an additional 1 mm."

[0097] The trained ML classification model can then be used to analyze the user's text instructions related to the most recent treatment. For each text instruction, the machine learning agent can classify the text instruction as belonging to one of two categories (positive or negative). Thus, a dataset (e.g., a list) of the user's text instructions can include the category of each text instruction, classifying it as positive or negative. The resulting classified dataset can be labeled and validated. Figure 8 The table in (Table 3) shows examples of text instructions from different users (“preference texts”) that have been classified into positive or negative categories by the machine learning agent.

[0098] The machine learning agent can be modified to improve the classification of user text information to match (positive) specific key preferences, or not match (negative) specific key preferences. For example, in order to achieve better classification, one or more filters can be applied to the analyzed user text instructions to train and apply to instruction text from a specified user. For example, the text can first be converted to all lowercase (or uppercase) text; the text can be filtered to ignore or remove certain characters or symbols, such as numbers, punctuation, common words (e.g., hi, thank you, goodbye, etc.); the text can be split into phrases (e.g., by the end of the line, the end of the sentence, punctuation, etc.).

[0099] In general, any suitable machine learning model can be used for text classification, including but not limited to: random forest, bag of words, and weod2vec. For example, random forest or random decision forest is an ensemble learning method for classification, regression and other tasks, which builds a large number of decision trees at training time and outputs the class as the class mode (classification) or mean prediction (regression) of each tree. Random decision forest corrects the habit of decision trees to overfit to their training set.

[0100] The bag-of-words model is a simplified representation used in natural language processing and information retrieval (IR). In this model, a text (e.g., a sentence or a document) is represented as a bag (multiset) of its words, disregarding grammar or even word order, but maintaining diversity. The bag-of-words model is often used in document classification methods, where the occurrence (frequency) of each word is used as a feature to train a classifier.

[0101] Word2vec can refer to a group of related models for producing word embeddings. These models are shallow two-layer neural networks that are trained to reconstruct the linguistic context of words. Word2vec takes a large text corpus as its input and produces a vector space (usually hundreds of dimensions) in which each unique word in the corpus is assigned a corresponding vector in the space. The word vectors are positioned in the vector space so that words that share a common context in the corpus are close to each other in the space.

[0102] The ML agents described herein can simplify and increase the speed of retrieving new text instructions at no additional cost, and can provide significant improvements in text matching using ML agents.

[0103] Back to Figure 7 , the ML agent 703 can be included in the workflow as at least part of a text classification system that can be used to automatically generate a list of candidate key preferences from the user's text instructions 101. For example, in some variations, the ML agent can be configured to use a modified bag of words model to train key preferences and find key preferences from user recommendation texts. As described above, the identified candidate key preferences (inferred key preferences) can be manually or semi-automatically reviewed or evaluated to confirm that the identified terms from the text instructions are related to the key preferences. The classification model can be improved by using, for example, a convolutional neural network (CNN) for text classification problems. In some variations, the ML agent may be required to work without a data set for training and testing, so data labeling can only be performed after text classification has been completed. For each possible text match to help automatically generate a treatment plan, only a small set of text instructions (manually collected in some cases) can be used. Therefore, when the data set is very small, unbalanced and "dirty", the trained model is usually very inefficient. This can be addressed by using clinical analysis techniques that provide each user with the most regular recent clinical behavior and personal data corresponding to actual user (or user group) data, such as the most recently configured treatment plan and instruction set within a defined or determined time period with a high confidence level. Fig. 9 An analysis of the key preference term "Start precise cutting at stage #" for a specific user from a historical treatment data set including treatment plans and user instructions for a large number of users collected over a two-year period is shown. Fig. 9 A time period extending from a first date 715 days ago to a second date (eg, the current date) is shown. Fig. 9 The vertical line 909 in represents the date after which the user has shown a clear clinical pattern (439 days ago). The precise cutting stage of the most recent case is shown on the left side of the vertical line, and the older cases are shown on the right side of the vertical line. Therefore, the ML agent can more effectively consider all users' text preferences as positive samples only from the time the current treatment preference is established (e.g., from 439 days ago). Therefore, by limiting the data to only current clinical behaviors corresponding to key preferences, the ML agent can be trained and can operate more reliably to determine positive categories, while the negative data set can be formed by the preferences of users who do not have any behavioral patterns (stage confusion). This method is useful for classifying the text of an entire user, which can include many different text instructions (usually equivalent to at least one sentence). However, it may be more helpful to determine which short instructions correspond to user intents related to the presumed key preferences (targets). Therefore, the ML agent can use a bag-of-words ML model that matches certain words with values ​​corresponding to their importance in the calculation process of text categories. As an example, Fig. 10A An example of text instructions (shown as text units or words with numerical values ​​above them) labeled by different values ​​is shown. A larger numerical value may correspond to a greater importance of the word to a text category, where the user instruction requests a target preference, such as "start precise cutting at stage #". For simplicity, if the entire text contains the word, these values ​​may be referred to as conditional probabilities that the entire text corresponds to the target category. In the following description, these conditional probabilities may be labeled:

[0104] P(class + │word)

[0105] The probability that a sentence expresses the user's target preference can be calculated as the sum of the following:

[0106]

[0107] in,

[0108] P(class + │word i ) is the conditional probability that the user's entire text has an instruction related to the target preference if the user's entire text contains the word numbered i. N is the total number of unique words in the entire dataset.

[0109] I(word i∈sentence) is an indicator function that shows the sentence contains the i-th word. The same probability can be obtained based on the entire instruction / sentence rather than a single word through the bag-of-words model, as Fig. 10B shown.

[0110] Therefore, according to this method, the conditional probability of each unique instruction in the dataset can be calculated:

[0111] P(class + │ instruction j )

[0112] where j < M is the total number of unique instructions.

[0113] To be able to calculate the probability of any sentence based on the included words, the following model can be fitted:

[0114]

[0115] Therefore, the probability of any sentence can be evaluated as:

[0116]

[0117] Then, these estimated probabilities can be used to determine whether the text being examined includes critical preferences based on this ML agent using the bag-of-words.

[0118] Alternatively or additionally, text retrieval can include one or more text matching techniques (e.g., Damerau-Levenshtein distance metric) that perform similarity retrieval to find a match between the terms in the user text instruction and one or more critical preferences. As described above, text instructions that match known critical preferences (e.g., critical preferences corresponding to clinical behaviors) can apply known text pattern techniques. In some variations of the methods and apparatuses described herein, text similarity can be used in a retrieval algorithm based on, for example, the Damerau-Levenshtein distance metric to find text instructions of a user or group of users corresponding to critical (e.g., clinical) preferences. A similarity retrieval algorithm based on the Damerau-Levenshtein distance (D-L or edit distance) metric can find sentences in the input text that are similar to a given reference. For example, a similarity threshold can be specified between 0 and 100%. 100% means the technique should match exactly the same sentence (ignoring letter case and word order in this example), while 0% means completely different sentences will be matched as similar. Generally, a higher threshold can provide more accurate results, while a lower threshold can give more positive matches but may also significantly increase the error rate (false matches).

[0119] Fig.11is a table showing an example of text similarity using reference key preferences when applying similarity retrieval (e.g., edit distance (e.g., DL distance) metric) as all or part of text matching / text retrieval. The use of edit distance can increase the number of key preferences identified, especially when used with a machine learning (ML) method (e.g., one of those methods described above), by up to 20%. In addition, the edit distance metric can also significantly reduce the amount of time to create initial datasets and text samples compared to manual text tagging. This improvement is even greater when the language of the tagged text is not the native language of the person performing the tagging.

[0120] Fig.12 is a workflow that includes using an edit distance metric as part of the text matching section 1205 (e.g., similar to Figure 1 and Figure 7 ), the text matching section 1205 includes both a machine learning (ML) agent 1202 and a similarity search 1204. As described above, at least two types of workflows may include the identification of key preferences. The first type of workflow may include identifying key preferences to initially construct a set of user or user group preferences, which may include training and testing text samples that can be used for machine learning (ML) agents. The trained agent can be used to analyze the text instructions of users over a long period of time. The result can be a list of text instructions (presumed key terms) with a high probability (reflecting the actual key preferences) corresponding to clinical preferences. A technician with appropriate clinical knowledge can then manually or semi-automatically mark and / or verify the found instruction text as a key preference. The obtained key preferences, user identification, and corresponding clinical behavior can form a set of user or user group preferences, which can be used as an initial data set for a similarity search technique (e.g., a DL distance metric) to expand or refine the list of found instructions. Similarity search techniques can help find the following text instructions, which otherwise may be skipped by ML techniques due to keyword misspellings, different word orders, and inaccurate algorithms.

[0121] In addition, similarity retrieval techniques can be effectively used for small amounts of initial text instructions. For example, when the number of initial text instructions is insufficient to train an ML agent, similarity retrieval can be used as a workflow separate from ML (e.g., without an ML agent). This can be very helpful for identifying key preferences of users who write instructions in different languages ​​or combinations of languages ​​(e.g., Spanish, French, Japanese, etc.). Fig.12 The rest of the workflow shown in can be used with the above Figure 1 and Figure 7The found instructions must be manually verified before being added to the list associated with the clinical preference text instructions. This list can be used again as the initial data set for the search algorithm whenever the search gives new results.

[0122] The similarity search technique described above is a DL distance metric that can be applied to individual words. As described above, the text to be checked by similarity search can be first prepared, which can improve the accuracy of the search. For example, in some variations, the input text may be split by the end of a sentence (dot) or the end of a line (\n); all numbers may be replaced with placeholders; the text may be normalized by case (e.g., all lowercase or uppercase); and all special symbols and punctuation may be removed.

[0123] Therefore, instead of directly measuring the distance between the entire sentences, or in addition to measuring the distance between the entire sentences, the similarity retrieval (e.g., DL distance determination) techniques described herein can use a word-by-word distance measure. If the word similarity is greater than a specified threshold, they are considered a match. The overall sentence similarity can be calculated using the following formula:

[0124]

[0125] Among them, Lsim i is the length of the i-th word marked as similar, Ldiff i is the length of the ith word marked as different, and Lall is the sum of the lengths of all words. If the resulting similarity is greater than a threshold amount, the word (eg, or phrase, eg, sentence) will be added to the verification list. Fig.14 An example of semantic similarity is shown, where the key preference "no attachment in overcorrection aligners" is compared to the user's instruction text (written as "Remove attachments at overcorrection"). The changes required to make the two statements identical are shown below (e.g., 3 substitutions, 9 insertions, 4 deletions, 1 transposition), from which a similarity score can be estimated as shown.

[0126] Therefore, DL distance is a string metric used to measure the edit distance between two sequences. Informally, the Damerau-Levenshtein distance between two words is the minimum number of operations (including the insertion, deletion, or substitution of a single character, or the transposition of two adjacent characters) required to change one word into another. Using this technique, the similarity between search results can be determined. Fig.13Table (Table 4) is shown, which shows the clinical behavior of the word matching using DL distance, unique user identifier, text matching with reference from its text instruction and current instruction. Technicians (or users in some cases) can accept or reject the suggestions shown. The text matching described herein can be improved by applying word-by-word similarity retrieval. Therefore, each word from the user's text instruction can be matched with each word from the reference (e.g., inferred key preference). In addition, similarity retrieval can limit the size of the reference key preference (e.g., character length or size length) to less than the matching threshold, for example, limiting similarity retrieval to compare the following fragments of the text instruction (e.g., single sentences, phrases), etc.), which are between about 50% to about 165% of the length of the reference (e.g., inferred key preference) (e.g., between 35% and 185%, between 40% and 180%, between 45% and 170%, between 50% and 165%, between 55% and 160%, etc.).

[0127] Fig.15 An example of a method for implementing a similarity retrieval technique such as DL distance is shown. Fig.12 As shown, this technique can be integrated into a workflow (e.g., a method for automatically generating a treatment plan). Fig.15 , a set of user or user group preferences is constructed using similarity retrieval. In this example, the device or method can obtain all instructions 1507 for each user 1501 (e.g., each ClinID) from a clinical database 1503 to generate data 1505 showing the instructions linked to each user. The method or device can then apply a text filter 1509 to format the text of the instructions (e.g., all lowercase or uppercase, remove punctuation, etc.). The prepared text 1511 can then be processed to perform a similarity retrieval 1515 as described above, such as using a similarity retrieval technique (e.g., DL distance) to compare the inferred key preferences of each user with all or part of the final instructions and / or clinical behavior from a historical treatment data set (e.g., a reference database) 1513 as described above. Results with the highest similarity above a similarity threshold or predicted level (e.g., >X% matches, where X is 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, etc.) may be considered true matches and may be added to the user or user group's set of preferences 1517. The search may be repeated with new references to build and / or update the user or user group's set of preferences.

[0128] Typically, a machine learning agent and / or similarity retrieval agent for identifying key preference languages ​​may be used to generate a set of user or user group preferences, as described above, and in Figure 7 and Fig.12is shown in the workflow of . Therefore, these agents (which may be referred to as engines in this article) can be used separately or together. In particular, the similarity retrieval agent can be used to improve the behavior of the machine learning agent, for example by expanding the positive (or negative) set used to train the machine learning agent. Either or both of the machine learning agent (or a new machine learning agent) and the similarity retrieval agent can be used to identify or confirm a match between the user's current text instruction and one of the key preferences in the user's or user group's key set. Different (and in some cases more stringent) thresholds for confirming a match can be applied to confirm a match between the user's text instruction for a new case and the set of user or user group preferences because the user is compared with their own language. Alternatively, in some variations, the text may have to match completely or almost completely.

[0129] Statistical Preference Analysis

[0130] In some cases, a set of user or user group preferences may be generated, expanded, and / or updated by using statistical analysis to identify users with consistent clinical behaviors and / or text instructions, even when those text instructions do not match candidate key preferences.

[0131] Therefore, in some variations, user behavior may be used to identify clinical behaviors and corresponding instructions (key preferences), alternatively or in addition to retrieving user text instructions to automatically or semi-automatically identify intended clinical instructions for generating one or more treatment plans as described above. For example, user-specific key preferences (related to clinical behaviors) may be automatically or semi-automatically determined by detecting statistically consistent clinical behaviors from the user's past behaviors. Generally, apparatus and methods including a statistical preference analysis agent (or statistical preference analysis engine) may identify user-specific clinical behaviors and corresponding key preferences (e.g., text instructions) by examining stored treatment data for a specified user over time from a historical treatment data set. When there are statistically consistent behaviors (e.g., treatment instructions and / or clinical behaviors) for a specified user, the user, the user's key preferences, and the corresponding clinical behaviors may also be added to a set of user or user group preferences. As Fig.16A As shown, this process can be executed in parallel with the above-mentioned text matching process.

[0132] The use of this statistical analysis (e.g., use of a statistical preference analysis agent) can improve the accuracy of the automatic interpretation of user instructions and can expand the number of users to whom the automatic interpretation can be applied. These methods and apparatus can also identify key preferences of users who do not explicitly provide text instructions for their treatment, or do not provide text instructions in a standard manner. Thus, these methods can improve and expand the operation of machine learning agents and / or similarity retrieval agents. Finally, these methods and apparatus can also or alternatively be applied to users with non-English text instructions and protocols of in-house technicians, and can verify previously identified user text instructions and associated clinical behaviors.

[0133] For example, a statistical preference analysis agent may determine whether a given user is highly consistent across multiple cases in the user's text instructions in response to a request to form a treatment plan for a patient. The method or apparatus may set a threshold (or more than one threshold), but may otherwise look for instructions or requests that are substantially the same across multiple cases, above a certain threshold for comparison.

[0134] Typically, there may be common clinical behavior patterns during the treatment plan for all users, which relate to different treatment phases. The most frequent clinical behavior patterns (which may be all key preferences (e.g., text instructions) or a subset of key preferences in a set of user or user group preferences) can be used as a starting point for performing statistical analysis. For example, a set of user or user group preferences may include the output of a machine learning agent and / or the output of a similarity retrieval agent, identifying the user, the corresponding key preferences for the user and the clinical behavior. A collection of historical treatment data (e.g., a list of users, including users for whom the machine learning agent and / or similarity retrieval agent did not identify any or many of their key preferences) can be analyzed separately for each user to determine the individual behavior detection purposes of each user.

[0135] In one variation, previously accepted user treatment plans for a particular user are studied to identify consistent clinical behavior patterns. One or more indicators (e.g., the frequency of treatment options and their standard deviation) are calculated. The analysis may provide indicator thresholds that indicate partial clinical behavior. Users with stable clinical behavior patterns may be assumed to be candidates for further common text preference analysis. The method may provide an inexpensive way to generate or expand a set of user or user group preferences (particularly when users use irregular key preferences (keywords)) to indicate the user's clinical preferences.

[0136] Therefore, the actual (or recent) detection of the user's clinical behavior can be an additional step to or before the statistical analysis. This step can assume a date after the user specified the clinical preference. Therefore, the user's clinical behavior may appear consistent from this assumed date to the present. Statistical treatment analysis can effectively detect this clinical behavior survey.

[0137] A direct application of the technique (or implementation of a statistical preference analysis agent) may look at the statistical occurrence of specific key preferences and any corresponding clinical behaviors in a user's text instructions; alternatively or additionally, the method and / or statistical preference analysis agent may examine the statistical occurrence of one or more clinical behaviors (including those clinical behaviors corresponding to key preferences that have been identified for other users in a user or user group preference set). If the variation in key preferences and / or clinical behaviors specific to a particular user (e.g., clinical behaviors that are common to other users and in a user or user group preference set) is very low, then the standard deviation of the mean of the behavior is equal to or above a threshold value (e.g., within Y% of the mean, where Y is 5, 10, 15, 20, 25, 30, 35, 40, 50, 60, etc.).

[0138] Therefore, if Fig.16A As shown in , statistical preference analysis (or a statistical preference analysis agent) may be included in a workflow for generating a set 109 of user or user group preferences. Fig.16A yes Figure 1 , Figure 7 and Fig.12 A modification of the workflow shown for generating a set of user or user group preferences. Fig.16A , statistical treatment analysis can be performed in parallel with the use of text matching (text retrieval 105') of the collected text instructions 101 as described above, including identifying and reviewing candidate key preferences 1606 for certain users, and manually or semi-automatically organizing the results 107, which are then added to a collection of user or user group preferences 109. Statistical preference analysis can be performed by identifying one or more clinical behaviors 1604 for treatment and / or user-specific key preferences, and determining how consistently the user uses them based on historical treatment data (clinical behaviors 1608). After manual or semi-automatic verification 107, 107', the consistently applied clinical behaviors and their corresponding user-specific key preference language (as in a parallel text matching path) are added to the collection of user or user group preferences together with the user identification.

[0139] In some cases, particularly for historical treatment data extending over a long period of time, users may change their practice one or more times, e.g., from a first consistent practice to a second consistent practice. This can be accounted for by examining clusters of behaviors that can be represented numerically (e.g., treatment phases may change for behaviors associated with actions taken at a particular treatment phase).

[0140] For example, in a dataset of historical treatment data extending over a period of 1.6 years, it can be assumed that users may change their clinical preferences periodically at least a few times since the beginning of the period. In addition, for each clinical feature, a user may have more than one clinical behavior at the same time, such as performing IPR or extraction at different stages depending on the patient's age or product type. In general, the situation may be very complex and it is difficult to find clear definitions. In this case, the numerical preferences (such as the stage to perform any clinical feature, IPR, extraction) may not have small deviation values, on which the investigation of the original statistical data is based.

[0141] In this case, it may be helpful to define the date from which the user's recent personal behavior was most intense (e.g., most consistent), and / or identify one or more behavioral patterns that cover all or nearly all of the user's treatment cases.

[0142] Some clinical features may be referred to as numerical features because they can be easily quantified. For example, numerical features may include the stage at which an IPR was initiated (stage number), the stage at which an attachment was removed (stage number), etc. Non-numeric features for which it may be difficult to define a distance metric may include, for example, text instructions from a review.

[0143] Numerical features can be processed in multiple dimensions (e.g., providing multiple clusters). For example, a distance metric can be used. A distance metric for numerical features can use the Euclidean norm. For clustering purposes, the [feature] x [timeline] plane can be considered. A two-dimensional (2D) distributed Gaussian mixture clustering model can be applied. It can be assumed that users cannot change their behavior more than a certain maximum number of times within a limited time period (e.g., no more than 5 times in 1.5 years). The model may be applicable to various clusters, where each cluster can be regarded as a consistent preference of the user. Therefore, as a result, there may be, for example, 5 configurations with different numbers of clusters. A configuration can be selected based on the minimum average of the standard deviations within each cluster in the configuration.

[0144] The most recent user preference can be selected from the selected configuration as a cluster. Based on the clustering results of the selected configuration, for each cluster, the earliest case can be found as the "since_date" attribute of the cluster, and the cluster with the "since_date" closest to (today) can be selected. The "since_date" of the selected cluster can be regarded as the last clinical preference change date. All cases since this date should be considered to include the actual current user preference. Therefore, for all these cases, the next clustering model can be applied. Cases from the most recent "since_date" can be processed by the KMenas clustering model. It can be assumed that the user cannot have more than four different behavior patterns and the same preference at the same time, so the KMeans model can be trained 3 times and the optimal number of clusters can be selected based on the overall / average standard deviation value. Additional cluster filtering can be based on the size of the cluster and the standard deviation around the cluster centroid. Therefore, the target results can include: the clinical preference start date and the centroid of the cluster, which are the values ​​of the target numeric feature.

[0145] Fig.17 An example of the results of digital features of clinical behavior is shown. Vertical line 1709 is the found start date of the actual current clinical behavior. The final cluster is filtered using a threshold of standard deviation within the cluster of <0.5. The example shows a single user ("Doctor A") as a [Feature] x [Timeline] graph. In this example, Doctor A has a consistent preference for digital clinical features in stage 13 since the vertical line date. In this example, the selected time period has a single clinical feature (13, corresponding to the stage of an action (e.g., interproximal removal, etc.) performed, including 423 cases; since the start of the identified cluster (vertical line 1709), the feature has been consistently applied to 98.58% of all cases, and the standard deviation is 0.0.

[0146] Fig.18 A second example of another user ("Doctor B") is shown. In this example, Doctor B has consistent preferences for multiple numeric clinical features: 7 and 13. Fig.18 A [Feature] x [Timeline] graph is shown for Doctor B. In this example, statistics for the selected time period 1809 can be determined for two clusters. The first cluster is at clinical feature 13, which includes 64 cases (17.63% of the number of cases since the beginning of time period 1809, with a standard deviation of 0.0); the second cluster is at clinical feature 7, which includes 266 cases (73.28% of the number of cases since the beginning of time period 1809, with a standard deviation of 0.23).

[0147] In some cases, features may be text features that are not immediately suitable for numerical representation (e.g., represented as dates) for examining different clusters. Therefore, it may be helpful to understand what users write most often in comments and text instructions. It can be assumed that the text instructions are related to the user's clinical preferences. These preferences can change, so it may be useful to use recent actual text instructions. These instructions can be filtered and normalized: remove common phrases for analysis (hello, thank you, etc.); set all letters to the same case (e.g., capital letters in phrases are variable and should not have any impact on the analysis results); remove enumeration and list markers from the instruction sequence; and / or split long instructions at line breaks and / or periods. Text instructions may be difficult to estimate distance metrics between variations due to sensitivity to phrase length differences or computational costs of the relationship between each and each.

[0148] In some variations, a phrase metric calculation may be estimated. Since the frequency of occurrence of text instructions may be low, the analysis period may be divided into weeks, and the frequency of each text instruction in that week may be calculated. The instruction frequency may be weighted by the weight of the week. Thus, for each instruction, there may be a vector of weekly values. In addition, an instruction frequency curve may be linearly approximated using the values ​​of the vector. The integral of the curve may be used as the instruction metric. The apparatus or method may then determine a combination of instructions whose sum of metrics is the largest; the number of instructions in such a combination may be limited to a maximum number (e.g., the top 3, top 10 instructions, etc.).

[0149] A tolerance period can be used to address this problem. The tolerance period can be several weeks, during which a set of instruction combinations is not changed even if the sum of their metrics is less than the maximum sum of the current metrics. This can be useful when there are frequent instructions for a specific case type (which is not very common). As described above, a tolerance period can be used to form a first set of instructions. The metrics of the instructions can be calculated during the tolerance period, and the original metric ranking can be used to select the initial set of instructions. Fig.19 An example of a block diagram describing an example of a text feature definition process is shown. Fig.19In, the illustrated method shows a method of limiting text features by iteratively determining the maximum metric sum. In this example, the calculation period is increased by one week each time, and all indicators are recalculated for the new calculation period. If the metric sum of the selected instruction is the maximum value, the calculation period will be increased and the process will move forward. If the metric sum of the current instruction has not reached the maximum value and the remaining tolerance period is positive, the tolerance period will be reduced, but the calculation period will be increased and the process will continue. If the tolerance period is zero or negative, the process will end: Assuming that the preference changes on this date, the date is calculated as: now()-(current_calculation_preiod-initial_tolreance_period)*7days. In this example, the current tolerance period is 5 weeks.

[0150] The statistical preference analysis methods described herein (and the statistical preference analysis agents used to perform them) can be used to sharpen data segments (e.g. Fig.17 , where only the more relevant recent data is used). This can greatly reduce the impact of statistical noise.

[0151] Example

[0152] As described above, described herein are methods and apparatus for creating a user preference database (and / or for automatically creating a treatment plan). For example, Fig. 20 A method 2000 for creating a user preference database is shown, which includes first identifying a reference key preference text statement corresponding to a clinical behavior for treating a patient's teeth 2001. The reference key preference text statement is a key preference as described above, which can be used as a reference when reviewing a user text instruction set. The user-specific text instruction set can be a database or set of historical treatment data (or derived from a database or set). The method can then include processing such user-specific text instruction set 2003, which includes a plurality of dental treatment instructions each associated with a specific user. The set can include data from a plurality of different users. Thereafter, the user-specific text key preference statement can be identified from the user-specific text instruction set by matching the reference key preference text statement with the user-specific text instruction from the user-specific text instruction set within a specified probability range 2005. The user-specific text key preference statement can be any key preference in the text of a specific user (e.g., perform IPR at stage X, extract teeth at stage Y, etc.).

[0153] Matching refers to text matching and can be performed by any of the methods described above, including by machine learning agents, by similarity retrieval, and / or by a combination of these 2007.

[0154] Thereafter, the user preference database may be updated with a cross-reference of the specific user, key preferences (user-specific textual key preference statements), and an identified match of associated clinical behaviors for treating the patient's teeth 2009. Optionally, the method of generating a user preference database may be included as part of a method for automatically generating a dental treatment plan, which may include the steps of: receiving a description of a patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and automatically generating a dental treatment plan using the user preference database.

[0155] Fig.21 Another example of a method (e.g., a computer-implemented method) for automatically generating a treatment plan using a user preference database is shown. Fig.21 In the method, the method may first create or receive access to a user preference database 2101, wherein the database includes a plurality of user identifiers, one or more user-specific text key preferences corresponding to each user identifier, and clinical behaviors corresponding to each user-specific text key preference. A computer executing the method may then receive a request for a new treatment plan (e.g., receiving a description of a patient's dentition 2103, and receiving user-specific text instructions 2105 for performing an orthodontic procedure on the patient). Finally, a dental treatment plan may be automatically generated 2107 by interpreting the user text instructions using the user preference database to match one or more user-specific text key preferences with one or more phrases in the user-specific text instructions, and incorporating the clinical behaviors corresponding to each matched user-specific text instruction into the dental treatment plan.

[0156] Fig. 22 Another example of a method for creating a user preference database is shown. Fig. 22In the present invention, a user preference database can be created by performing a statistical preference analysis to identify key preference text statements associated with a single user and corresponding clinical behaviors for treating a patient's teeth from a user-specific text instruction set including a plurality of dental treatment instructions, each of which is associated with a specific user 2201. In some variations, this can include determining a time period during which the statistical analysis is performed (e.g., a recent duration during which the clinical behaviors are consistent). The statistical preference analysis can be performed as described herein and is typically performed for a single user at a time. In the case where the clinical behavior of a single user is stable, for example, where the same clinical behavior is repeatedly used over time with very low changes, the corresponding key preferences (e.g., key preference text statements) can be identified from the text instructions of the same user, and the clinical behavior, user identifier, and corresponding key preferences can be added to the user preference database 2205. As described above, any of these methods can optionally include receiving a description of a patient's dentition, receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient, and automatically generating a dental treatment plan 2207 using the user preference database.

[0157] Fig.23 2300 is a diagram showing an example of an automatic treatment planning system 2300, which may include a treatment planning engine 2302 and a user preference engine 2304 as part thereof, for generating, modifying and updating a set of user or user group preferences (equivalent to a user preference data store or a user preference database). As shown, the modules of the treatment planning system 2300 may include one or more engines and data stores. The computer system may be implemented as an engine, as part of an engine or by multiple engines. As used herein, an engine includes one or more processors or a portion thereof. A portion of one or more processors may include a certain portion of hardware (which is less than all the hardware including any given one or more processors) (such as a subset of registers), a portion of a processor dedicated to one or more threads of a multi-threaded processor, a time slice in which a processor is fully or partially dedicated to implementing a portion of the functions of the engine, etc. In this way, the first engine and the second engine may have one or more dedicated processors, or the first engine and the second engine may share one or more processors with each other or with other engines. Depending on implementation-specific or other considerations, the engine may be centralized or its functions may be distributed. The engine may include hardware, firmware, or software embodied in a computer-readable medium for execution by a processor. The processor converts data into new data using implemented data structures and methods, such as described with reference to the accompanying drawings of this article.

[0158] The engine described herein or the engine by which the systems and devices described herein can be implemented can be a cloud-based engine. As used herein, a cloud-based engine is an engine that can use a cloud-based computing system to run applications and / or functions. All or part of the application and / or function can be distributed between multiple computing devices and does not need to be limited to only one computing device. In some embodiments, a cloud-based engine can execute functions and / or modules that an end user accesses through a web browser or a container application without having to install the functions and / or modules locally on the end user's computing device.

[0159] As used herein, "datastore" is intended to include repositories having any applicable data organization, including tables, comma-separated value (CSV) files, traditional databases (e.g., SQL), or other applicable known or convenient organizational formats. The data store may be implemented, for example, as software embodied in a physical computer-readable medium on a dedicated machine, firmware, hardware, a combination thereof, or an applicable known or convenient device or system. Data store-related components such as database interfaces may be considered "part" of the data store, part of some other system component, or a combination thereof, although the physical location and other characteristics of the data store-associated components are not critical to understanding the techniques described herein.

[0160] Data storage may include data structures. As used herein, a data structure is associated with a specific way of storing and organizing data in a computer so that it can be used efficiently in a given context. Data structures are typically based on the ability of a computer to obtain and store data at any location in its memory, which is specified by an address, a bit string that can itself be stored in the memory and manipulated by a program. Therefore, some data structures are based on calculating the addresses of data items using arithmetic operations; while other data structures are based on storing the addresses of data items within the structure itself. Many data structures use these two principles, sometimes combined in a non-trivial way. The implementation of a data structure typically requires writing a set of programs that create and manipulate instances of the structure. The data storage described herein may be a cloud-based data storage. A cloud-based data storage is a data storage that is compatible with a cloud-based computing system and engine.

[0161] The treatment planning system 2300 may include a computer-readable medium, a treatment planning engine 2302, a user preference engine 2304, which may include one or more machine learning engines 2306 (e.g., machine learning agents), similarity search engines 2308 (similarity search agents) and / or statistical preference engines 2310 (statistical preference agents), historical treatment data storage 2307, and user preference data storage 2305. One or more modules of the automatic treatment system 2300 may be coupled to each other (e.g., via Fig.23The example coupling shown) or coupled to Fig.23 The system may include a computer-readable medium, which may include any computer-readable medium, including but not limited to a bus, a wired network, a wireless network, or some combination thereof.

[0162] The automatic treatment planning system 2300 or any of its components may implement one or more automatic agents configured as, for example, machine learning agents / engines to learn the matching of texts by training as described herein. The treatment planning system may implement one or more automatic agents configured to manufacture an appliance or a series of appliances. Examples of appliances are described in detail in the following documents: U.S. Patent No. 5,975,893 and published PCT application WO98 / 58596, which are incorporated herein by reference for all purposes. Systems of dental appliances using the technology described in U.S. Patent No. 5,975,893 are available from Alane Technology Co., Ltd. of San Jose, California, under the trade name Invisalign system. Throughout the specification herein, with respect to dental appliances, the use of the terms "orthodontic appliance", "aligner" or "dental appliance" is synonymous with the use of the terms "appliance" and "dental appliance". For clarity, embodiments are described below in the context of using and applying appliances (more specifically, "dental appliances"). The aligner manufacturing can be part of a 3D printing system, a thermoforming system, or some combination thereof.

[0163] Any method described herein (including user interface) may be implemented as software, hardware, or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions that can be executed by a processor (e.g., a computer, tablet, smartphone, etc.), which, when executed by the processor, causes the processor to control the execution of any of the following steps, including but not limited to: displaying, communicating with a user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, issuing an alarm, etc.

[0164] When a feature or element is referred to herein as being "on" another feature or element, it may be directly located on the other feature or element, and / or there may also be intermediate features and / or elements. On the contrary, when a feature or element is referred to as being "directly on" another feature or element, there is no intermediate feature or element. It should be understood that when a feature or element is referred to as being "connected," "attached" or "coupled" to another feature or element, it may be directly connected, attached or coupled to the other feature or element, or there may be intermediate features or elements. On the contrary, when a feature or element is referred to as being "directly connected," "directly attached" or "directly coupled" to another feature or element, there is no intermediate feature or element. Although described or shown with respect to one embodiment, the features and elements described or shown in this way may be applied to other embodiments. Those skilled in the art will recognize that a reference to a structure or feature provided for "adjacent" another feature may have a portion overlapping or below an adjacent feature with an adjacent feature.

[0165] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the present invention. For example, unless the context clearly indicates, as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well. It should be further understood that the terms "comprises" and / or "comprising" when used in this specification specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0166] Spatially related terms, such as "under", "below", "lower", "over", "upper", etc., may be used herein to facilitate description of the relationship of one element or feature to another element or feature or multiple elements or features as shown in the accompanying drawings. It will be understood that spatially related terms are intended to include different orientations of the device in use or operation in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is reversed, the elements described as being "under" or "beneath" other elements or features will be oriented to be "over" other elements or features. Therefore, the exemplary term "under" can cover both orientations above and below. The device can be oriented otherwise (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly. Similarly, unless otherwise specifically stated, the terms "upwardly", "downwardly", "vertical", "horizontal", etc. are used herein for illustrative purposes only.

[0167] Although the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context indicates otherwise. These terms may be used to distinguish one feature / element from another feature / element. Therefore, without departing from the teachings of the present invention, the first feature / element discussed below may be referred to as the second feature / element, and similarly, the second feature / element discussed below may be referred to as the first feature / element.

[0168] Throughout this specification and the appended claims, unless the context requires otherwise, the word "comprise" and variations such as "comprises" and "comprising" mean that various components (e.g., compositions and devices including apparatus and methods) can be used together in methods and articles. For example, the term "comprising" will be understood to imply the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.

[0169] In general, any apparatus and method described herein should be understood to be inclusive, but all or a subset of components and / or steps may optionally be exclusive and may be expressed as "consisting of" or alternatively "consisting essentially of" various components, steps, sub-components or sub-steps.

[0170] As used herein in the specification and claims, including in the examples, and unless otherwise expressly stated, all numbers may be understood as if beginning with the word "about" or "approximately", even if the term does not appear explicitly. The phrase "about" or "approximately" may be used when describing an amplitude and / or position to indicate that the value and / or position described is within a reasonably expected range of values ​​and / or positions. For example, a numerical value may have a value of + / -0.1% of the value (or range of values), + / -1% of the value (or range of values), + / -2% of the value (or range of values), + / -5% of the value (or range of values), + / -10% of the value (or range of values), etc. Any numerical value given herein should be understood to include about that value or approximately that value, unless the context indicates otherwise. For example, if the value "10" is disclosed, "about 10" is also disclosed. Any numerical range described herein is intended to include all subranges contained therein. It should also be understood that when a value is disclosed as "less than or equal to" the value, "greater than or equal to" and possible ranges between values ​​are also disclosed, as appropriately understood by those skilled in the art. For example, if a value "X" is disclosed, "less than or equal to X" and "greater than or equal to X" (e.g., where X is a numerical value) are also disclosed. It should also be understood that throughout the application, data is provided in a variety of different formats, and that the data represents end points and starting points and ranges for any combination of data points. For example, if a specific data point "10" and a specific data point "15" are disclosed, it should be understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15 and between 10 and 15 are considered disclosed. It should also be understood that each unit between two specific units is also disclosed. For example, if 10 and 15 are disclosed, 11, 12, 13, and 14 are also disclosed.

[0171] Although various illustrative embodiments are described above, any of several changes may be made to the various embodiments without departing from the scope of the invention as described in the claims. For example, in alternative embodiments, the order in which the various described method steps are performed may generally be changed, and in other alternative embodiments, one or more method steps may be skipped altogether. Optional features of the various device and system embodiments may be included in some embodiments and not included in other embodiments. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be construed as limiting the scope of the present disclosure as set forth in the claims.

[0172] The examples and descriptions included herein show specific embodiments in which the subject matter can be practiced by way of illustration and not limitation. As mentioned, other embodiments can be utilized and derived therefrom so that structural and logical replacements and changes can be made without departing from the scope of the present disclosure. For convenience only, these embodiments of the subject matter of the invention may be referred to individually or collectively by the term "disclosed" herein, and it is not intended that the scope of the present application be actively limited to any single invention or inventive concept, if in fact more than one invention or inventive concept is disclosed. Therefore, although specific embodiments have been illustrated and described herein, it is predicted that any arrangement used to achieve the same purpose may replace the specific embodiments shown. The present disclosure is intended to cover any and all modifications or variations of various embodiments. After reading the above description, the combination of the above-mentioned embodiments and other embodiments not specifically described herein will be obvious to those skilled in the art.

Claims

1. A computer-implemented method comprising: Create a user preference database by: performing a statistical preference analysis to identify key preference text sentences associated with a single user and corresponding clinical behaviors for treating a patient's teeth from a user-specific set of text instructions comprising a plurality of dental treatment instructions, each of the plurality of dental treatment instructions being associated with a specific user, adding an entry to the user preference database, the entry linking the key preference text statement, a single user associated with the key preference text statement, and a corresponding clinical action for treating a patient's teeth; Receive a description of the patient's dentition; receiving patient treatment orders from a dental professional for performing an orthodontic procedure on the patient; as well as A dental treatment plan is generated using the user preference database.

2. The method according to claim 1, wherein: Generating a dental treatment plan using the user preference database includes: finding any user-specific text key preference statements from a user preference database associated with a dental professional; and incorporating clinical actions for treating the patient's teeth corresponding to any found user-specific key preferences into the dental treatment plan, wherein any found user-specific key preferences match phrases in the patient treatment instructions.

3. The method according to claim 1, wherein: Performing a statistical preference analysis includes confirming that clinical behavior for treating a patient's teeth remains approximately constant during a period of time.

4. The method according to claim 1, wherein: The statistical preference analysis includes identifying clinical behaviors for treating teeth of a patient and corresponding key preference text statements from clinical dental procedure instructions, wherein the clinical behaviors for treating teeth of a patient are statistically consistent during a period of time.

5. The method according to claim 1, further comprising: User-specific textual instructions including the plurality of dental treatment instructions are collected from the historical treatment data set, wherein each dental treatment instruction is associated with a clinical dental procedure associated with a single user.

6. The method according to claim 1, wherein: Performing the statistical preference analysis includes performing the statistical preference analysis after confirming that the clinical behavior for treating the patient's teeth is statistically consistent during a recent behavior detection period and after limiting the statistical preference analysis to a time period.

7. The method according to claim 1, further comprising: The key preference text statements and the corresponding clinical actions for treating a patient's teeth are manually verified before adding the key preference text statements and the corresponding clinical actions for treating a patient's teeth to the user preference database.

8. The method according to claim 1, wherein: The description of the recipient patient's dentition includes: a digital model of the recipient patient's dentition.

9. A computer-implemented method comprising: Create a user preference database by: collecting user-specific textual instructions including a plurality of dental treatment instructions from a historical treatment data set, the plurality of dental treatment instructions each being associated with a clinical dental procedure associated with a single user; identifying, from the clinical dental procedure instructions, clinical behaviors and corresponding key preference text statements for treating a patient's teeth that are statistically consistent during a period of time; adding an entry to the user preference database, the entry linking the key preference text statement, the single user, and a corresponding clinical action for treating a patient's teeth; Receive a description of the patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and A dental treatment plan is automatically generated using the user preference database.

10. The method according to claim 9, wherein: Automatically generating a dental treatment plan using the user preference database includes: finding any user-specific text key preference statements from a user preference database associated with a dental professional; and incorporating clinical actions for treating the patient's teeth corresponding to any found user-specific key preferences into the dental treatment plan, wherein any found user-specific key preferences match phrases in the patient treatment instructions.

11. The method according to claim 9, wherein: Performing the statistical preference analysis includes performing the statistical preference analysis after confirming that the clinical behavior for treating the patient's teeth is statistically consistent during a recent behavior detection period and after limiting the statistical preference analysis to a time period.

12. The method according to claim 9, further comprising: The key preference text statements and the corresponding clinical actions for treating a patient's teeth are manually verified before adding the key preference text statements and the corresponding clinical actions for treating a patient's teeth to the user preference database.

13. The method according to claim 9, wherein: The description of the recipient patient's dentition includes: a digital model of the recipient patient's dentition.

14. A computer-implemented method comprising: Create a user preference database by: collecting user-specific textual instructions including a plurality of dental treatment instructions from a historical treatment data set, the plurality of dental treatment instructions each being associated with a clinical dental procedure associated with a single user; identifying, from clinical dental procedure instructions, clinical actions for treating a patient's teeth and corresponding key preference text statements that are statistically consistent during a time period, wherein the time period includes a duration of a most recent cluster of clinical actions for treating a patient's teeth; adding an entry to the user preference database, the entry linking the key preference text statement, the single user, and a corresponding clinical action for treating a patient's teeth; Receive a description of the patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and A dental treatment plan is automatically generated using the user preference database.

15. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: Create a user preference database by: performing a statistical preference analysis to identify key preference text sentences associated with a single user and corresponding clinical behaviors for treating a patient's teeth from a user-specific set of text instructions comprising a plurality of dental treatment instructions, each of the plurality of dental treatment instructions being associated with a specific user, adding an entry to the user preference database, the entry linking the key preference text statement, a single user associated with the key preference text statement, and the corresponding clinical action for treating a patient's teeth; Receive a description of the patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and A dental treatment plan is automatically generated using the user preference database.

16. The system of claim 15, wherein: Automatically generating a dental treatment plan using the user preference database includes: finding any user-specific text key preference statements from a user preference database associated with a dental professional; and incorporating clinical actions for treating the patient's teeth corresponding to any found user-specific key preferences into the dental treatment plan, wherein any found user-specific key preferences match phrases in the patient treatment instructions.

17. The system of claim 15, wherein: The statistical preference analysis includes identifying clinical behaviors for treating teeth of a patient and corresponding key preference text statements from clinical dental procedure instructions, wherein the clinical behaviors for treating teeth of a patient are statistically consistent during a period of time.

18. The system of claim 15, wherein: The computer-implemented method further includes collecting user-specific text instructions including the plurality of dental treatment instructions from the historical treatment data set, wherein each dental treatment instruction is associated with a clinical dental procedure associated with a single user.

19. The system of claim 15, wherein: The computer-implemented method further includes performing the statistical preference analysis after limiting the statistical preference analysis to a time period after confirming that the clinical behavior for treating the patient's teeth is statistically consistent during a recent detection of the behavior.

20. The system of claim 15, wherein: The description of the recipient patient's dentition includes: a digital model of the recipient patient's dentition.

21. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: Create a user preference database by: collecting user-specific textual instructions including a plurality of dental treatment instructions from a historical treatment data set, the plurality of dental treatment instructions each being associated with a clinical dental procedure associated with a single user; identifying statistically consistent clinical behaviors and corresponding key preference text statements for treating a patient's teeth during a time period from clinical dental procedure instructions; adding an entry to the user preference database, the entry linking the key preference text statement, the single user, and a corresponding clinical action for treating a patient's teeth; Receive a description of the patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and A dental treatment plan is automatically generated using the user preference database.

22. A system comprising: one or more processors; a memory coupled to the one or more processors, the memory being configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: Create a user preference database by: collecting user-specific textual instructions including a plurality of dental treatment instructions from a historical treatment data set, the plurality of dental treatment instructions each being associated with a clinical dental procedure associated with a single user; identifying, from clinical dental procedure instructions, clinical actions for treating a patient's teeth and corresponding key preference text statements that are statistically consistent during a time period, wherein the time period includes a duration of a most recent cluster of clinical actions for treating a patient's teeth; adding an entry to the user preference database, the entry linking the key preference text statement, the single user, and a corresponding clinical action for treating a patient's teeth; Receive a description of the patient's dentition; receiving patient treatment instructions from a dental professional for performing an orthodontic procedure on the patient; and A dental treatment plan is automatically generated using the user preference database.

Images