Data Triage in Microscope Systems
The scientific instrument support system automates data selection and model promotion for microscopy, addressing inefficiencies and data access limitations, resulting in improved machine learning models and instrument performance.
Patent Information
- Application Number
- JP2024519583
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-10-01
- Filing Date
- 2022-09-30
- Publication Date
- 2024-11-06
AI Technical Summary
Existing methods for creating training data for machine learning models in microscopy are inefficient and prone to human error, requiring significant user time and computing resources, and are limited by proprietary data access restrictions.
A scientific instrument support system that includes a machine learning model to automatically triage and select useful data for training, allowing continuous learning and improving model performance without sharing proprietary data, by implementing data triage logic and model promotion logic to optimize the machine learning workflow at the customer level.
This approach reduces human effort and computing resources while ensuring data security, leading to more accurate and efficient machine learning models for microscopy, enhancing the operation of scientific instruments and associated processes.
Smart Images

Figure 2024540819000001_ABST
Abstract
Description
[Technical field]
[0001] (Related Applications) This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 63 / 251,351, filed October 1, 2021, the entire contents of which are incorporated herein by reference. [Background technology]
[0002] Microscopy is a technical field in which microscopes are used to better view objects that are difficult to see with the naked eye. Different areas of microscopy include, for example, optical microscopy, charged particle (electron and / or ion) microscopy, and scanning probe microscopy. Charged particle microscopy involves using a beam of accelerated charged particles as the illumination source. Types of charged particle microscopy include, for example, transmission electron microscopy, scanning electron microscopy, scanning transmission electron microscopy, and focused ion beam microscopy. [Brief description of the drawings]
[0003] The embodiments will be readily understood by the following detailed description taken in conjunction with the accompanying drawings, in which: To facilitate this description, like reference numerals refer to like structural elements, and in which: The embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which: [Figure 1A] FIG. 1 is a block diagram of an example scientific instrument support device for performing support operations, according to various embodiments. [Figure 1B] FIG. 1B is a block diagram of data triage logic of the assistance device of FIG. 1A according to various embodiments. [Figure 1C] 1B is a block diagram of model promotion logic of the support device of FIG. 1A according to various embodiments. [Figure 2A] 1 is a flow diagram of an exemplary method for performing an assist operation, according to various embodiments. [Figure 2B]2B is a flow diagram of an example method for performing a data triage operation as part of the method of FIG. 2A, according to various embodiments. [Figure 2C] 2B is a flow diagram of an example method for performing a model promotion operation as part of the method of FIG. 2A according to various embodiments. [Diagram 3] 1 is an example image including a number of identified features generated using a machine learning model of the image. [Figure 4] FIG. 1 depicts an example plot showing the number of features identified for each image in a set of images. [Diagram 5] FIG. 2 depicts an example plot showing feature regions for each image of a set of images. [Figure 6] FIG. 2 depicts an exemplary plot showing feature distances identified for each image of a set of images. [Figure 7] FIG. 1 illustrates an example plot showing the number of features identified per image in a set of images, where the example plot illustrates a failed machine learning inference. [Figure 8] FIG. 1 illustrates an example plot showing feature regions per image of a set of images, where the example plot illustrates failed machine learning inference. [Figure 9] FIG. 1 illustrates an example plot showing feature distances identified for each image in a set of images, where the example plot illustrates a failed machine learning inference. [Figure 10] FIG. 2 illustrates an example user interface for receiving selection criteria from a user, according to various embodiments. [Figure 11-1] FIG. 1 illustrates an example user interface for providing training results of a machine learning model, according to various embodiments. [Figure 11-2] FIG. 1 illustrates an example user interface for providing training results of a machine learning model, according to various embodiments. [Figure 12]FIG. 1 depicts an example plot showing performance metrics for a machine learning model, according to various embodiments. [Figure 13] FIG. 1 depicts an example plot showing performance metrics for a machine learning model, according to various embodiments. [Figure 14] FIG. 1 illustrates a graph of example performance metrics for multiple machine learning models, according to various embodiments. [Figure 15] FIG. 1 illustrates example model deployment criteria associated with a particular scientific instrument for enrollment with a machine learning server, according to various embodiments. [Figure 16] FIG. 1 illustrates an example user interface for manually deploying a machine learning model, according to various embodiments. [Figure 17] FIG. 1 illustrates an example of a graphical user interface that may be used in performing some or all of the assistance methods disclosed herein, according to various embodiments. [Figure 18] FIG. 1 is a block diagram of an example computing device capable of performing some or all of the scientific instrument-assisted methods disclosed herein, according to various embodiments. [Figure 19] FIG. 1 is a block diagram of an example scientific instrument support system capable of performing some or all of the scientific instrument support methods disclosed herein, according to various embodiments. [Figure 20] FIG. 1 is a block diagram of an example scientific instrument included in a scientific instrument support system, according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0004] Disclosed herein are scientific instrument support systems, as well as related methods, computing devices, and computer-readable media. For example, in some embodiments, a scientific instrument support apparatus for a scientific instrument (e.g., a charged particle microscope) is provided. The scientific instrument support apparatus may be implemented by a common computing device, which may be included in the scientific instrument or may be remote from the scientific instrument, and is configured to generate one or more identified features in images of a set of images acquired via the scientific instrument using a machine learning model. The scientific instrument support apparatus is also configured to determine whether the set of images meets one or more selection criteria. The scientific instrument support apparatus is also configured to assign the set of images, including the one or more identified features, to a training dataset in response to determining that the set of images meets the one or more selection criteria. The scientific instrument support apparatus is also configured to retrain the machine learning model using the training dataset. A method executed via the computing device for providing scientific instrument support is also provided.
[0005] The embodiments of scientific instrumentation disclosed herein may achieve improved performance over conventional approaches. For example, machine learning (ML) models (implementing one or more ML algorithms) have demonstrated improvements in target image localization, endpoint identification, as well as image quality enhancement, among others, compared to conventional methods. However, the performance of ML models often relies on training the models on images similar to those they will encounter during use or deployment. While large open datasets exist for human-scale features (e.g., humans, vehicles, animals), such datasets are unavailable for many microscopic features, for example, due to the specialized equipment, samples, and structures commonly found in microscopes. As a result, machine learning requires user-created microscopy data for training. However, this data may be used for proprietary or intellectual property (IP)-related applications that control access to the data (e.g., semiconductor microscopes), and in some implementations, such data is only available at the user's or customer's site, not to entities with the capabilities to build and train (including retrain) models using machine learning. In addition to access restrictions, having users create training data is inefficient. For example, having a user annotate large data sets consumes a large amount of user time and computing resources, can be prone to human error (e.g., the process is laborious and tedious), and is often not feasible given the amount of training data that needs to be annotated or labeled for use as training data. For example, many scientific instruments generate thousands of images per day. Thus, the embodiments disclosed herein provide improvements to scientific instrument technology (e.g., improvements in the computer technology that supports such scientific instruments, among other things).
[0006] The embodiments disclosed herein may provide improved machine learning models and associated data processing by such models compared to conventional approaches. For example, as described above, conventional approaches rely on users to create training data. However, as described above, such approaches suffer from many technical issues and limitations, such as inefficient use of computing resources to manually create such training data, and access control limitations associated with proprietary data in certain locations.
[0007] Various embodiments of the embodiments disclosed herein can improve upon conventional approaches to achieve the technical advantage of improved machine learning models, and thus improved operation of scientific instruments, through improved inference (e.g., identification of one or more features in image data) using models on acquired data, such as data acquired through a microscope, including a charged particle microscope (CPM). For example, a scientific instrument, such as a CPM, serves as a data source by generating output data, such as image data. This generated output can be used as a data source to improve machine learning performance by transferring the machine learning workflow to an end user (e.g., a customer), and this workflow can be repeated as new data (new image data from one or more CPMs) become available to maintain accuracy and reliability through the changing process. Among other things, this output can be fed back into the customer-level machine learning process by automatically selecting useful outputs (e.g., microscope images) as training data (i.e., determining which output data is useful for future machine learning), while continuing to protect data ownership and intellectual property rights. For example, embodiments described herein can automatically select useful output (e.g., one or more images, or one or more sets of images) generated by one or more scientific instruments (e.g., microscopes) and present the output for human review and annotation (e.g., through one or more user interfaces), where the annotated data can be used as training data to retrain a model, without requiring the user to have machine learning expertise. Automatically selecting such useful data avoids prompting the user to review and annotate all available data, resulting in a more efficient use of computing resources and therefore a more accurate machine learning model (e.g., improved effectiveness in edge cases that can be identified and presented to the user for human review and annotation).The selected training data is then used to improve the machine learning model to improve the operation and performance of the scientific instrument or a process involving the scientific instrument, such as improving sample preparation, sample processing (e.g., grinding), diagnostic imaging, machine configuration and operation, etc.
[0008] That is, the embodiments described herein automatically triage data output by one or more scientific instruments to generate training data for one or more machine learning models to optimize the value of such training data, while minimizing human effort and protecting access controls associated with such data. Such technical advantages are not achievable through routine and conventional approaches, and all users of a system including such embodiments can benefit from such advantages (e.g., by improving machine learning models to assist the user in performing technical tasks, such as endpointing). Thus, the technical features of the embodiments disclosed herein, as well as combinations of features of the embodiments disclosed herein, are clearly unique in the field of microscopy and other scientific instruments. As discussed further herein, various aspects of the embodiments disclosed herein can improve the functionality of the computer itself. For example, there are inference computers used via the scientific instruments to apply models to control or guide the operation of the scientific instruments, prepare or process samples, perform diagnostics, configure or calibrate the scientific instruments, etc. The computer and user interface features disclosed herein are applied to new analytical and technical techniques to modify the operation of the scientific instruments, not only through the collection and comparison of information, but also through the use of improved models acquired through improved learning processes. Thus, the present disclosure introduces capabilities that could not be performed by conventional computing devices or humans.
[0009] Thus, embodiments of the present disclosure may serve any of a number of technical purposes, such as controlling a particular technical system or process, determining how to control a machine from measurements, digital audio, image, or video enhancement, or analysis, or a combination thereof, etc. Among other things, the present disclosure provides technical solutions to technical problems including, but not limited to, generating machine learning models used to operate scientific instruments such as CPMs.
[0010] Thus, the embodiments disclosed herein provide improvements in scientific instrument technology (e.g., improvements in computer technology supporting scientific instruments such as microscopes, including CPM, among other improvements).
[0011] In the following detailed description, reference is made to the accompanying drawings which form a part of this specification, in which like numerals designate like parts throughout and which show, by way of illustration, embodiments which may be implemented. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.
[0012] Various operations may be described as multiple separate actions or operations, in the order that is most helpful in understanding the subject matter disclosed herein. However, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations may not be performed in the order presented. The operations described may be performed in a different order than in the described embodiment. Various additional operations may be performed and / or described operations may be omitted in additional embodiments.
[0013] For purposes of this disclosure, the phrases "A and / or B" and "A or B" mean (A), (B), or (A and B). For purposes of this disclosure, the phrases "A, B, and / or C" and "A, B, or C" mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Although some elements may be referred to in the singular (e.g., a "processing device"), any suitable element may be represented by multiple instances of that element, and vice versa. For example, a set of operations described as being performed by a processing device may be implemented with different ones of the operations performed by different processing devices.
[0014] This description uses the phrases "an embodiment," "various embodiments," and "some embodiments," each of which may refer to one or more of the same or different embodiments. Furthermore, terms such as "comprising," "including," and "having" used with respect to embodiments of the present disclosure are synonymous. When used to describe a range of dimensions, the phrase "between X and Y" represents a range that includes X and Y. As used herein, an "apparatus" may refer to any individual device, a collection of devices, a portion of a device, or a collection of portions of devices. The drawings are not necessarily drawn to scale.
[0015] FIG. 1A is a block diagram of an exemplary scientific instrument support module 1000 apparatus for performing scientific instrument support operations, according to various embodiments. As a non-limiting example, the scientific instrument support module 1000 is described herein as supporting CPM, and thus is also referred to herein as the "CPM support module 1000." The data triage, model promotion, or both described herein are applicable to various types of scientific instruments that employ machine learning models to generate inferences (e.g., diagnoses inferred from image data, such as one or more features identified in an image), and the embodiments described herein are not limited to CPM support. For example, the data triage and model promotion described herein, performed by the CPM support module 1000, may be used in cryo-electron tomography applications, gene sequencing applications, and other microscopy and imaging applications that use inferences from machine learning models.
[0016] CPM assistance module 1000 may be implemented by circuitry (e.g., including electrical and / or optical components) such as a programmed computing device. The logic of CPM assistance module 1000 may be included in a single computing device or distributed across multiple computing devices in communication with each other as needed. An example of a computing device in which CPM assistance module 1000 may be implemented, alone or in combination, is discussed herein with reference to computing device 4000 of FIG. 18, and an example of a system of interconnected computing devices in which CPM assistance module 1000 may be implemented across one or more of the computing devices is discussed herein with reference to scientific instrument support system 5000 of FIG. 19.
[0017] As described in more detail below, the CPM assistance module 1000 implements an ML workflow that uses previous inferences generated by a machine learning model to improve future inferences generated by the model, and the CPM assistance module 1000 performs automated data triage to identify what and how previous inferences to include in the ML workflow to efficiently create effective machine learning models for customers. The ML workflow may include collecting data, generating training datasets, retraining models, testing and validating models (when retrained), promoting and deploying models, or combinations thereof. The CPM assistance module 1000 may repeat the ML workflow as new data becomes available to implement a continuous learning workflow, thereby improving the machine learning model based on incoming data (e.g., from one or more CPMs), which in turn improves the performance of the CPM and other envisioned scientific instruments and processes (e.g., accurate sample preparation via the CPM). As used herein, "continuous" learning or a "continuous" workflow generally means that a machine learning model undergoes a repeated training process (i.e., retraining), for example, when new data becomes available or other triggering events occur. This repeated training includes performing automated data triage, the automated nature of which requires limited user intervention and user experience or expertise in machine learning processes, thereby allowing the learning workflow to be implemented at a customer or client level (e.g., an owner or operator of one or more scientific instruments, such as one or more CPMs) while adhering to data controls or access restrictions.
[0018] For example, as described in more detail below with respect to FIG. 20, the CPM generates a set of sample images, the set of images including one or more images. A machine learning model is applied to the set of images to determine one or more features (also referred to herein as "inferences") identified in the set of images. The one or more features identified may include stage detection, line indication end execution, device line termination, gliderator, image noise removal, or similar image features or artifacts. The inferences may be available for future training of the machine learning model. However, as discussed above, including all inferences in future training (without manual review or annotation) may reduce the effectiveness of the machine learning model, especially with respect to edge cases, and may require significant computing resources (e.g., memory and processing resources) given the amount of image data and associated inferences generated. However, having a user manually review, triage, and annotate all such available inferences (if necessary) requires cost-prohibitive overhead in many situations. Additionally, providing inferences available to third parties, such as parties skilled in machine learning, for use in further model training may limit the availability of training data due to some customers' reluctance or limitations in sharing image data, inferences generated from such image data, or both with other customers or organizations, and due to a lack of in-house experience in machine learning workflows and training, thereby limiting the data available for training and improving future machine learning models.
[0019] Thus, the CPM assistance module 1000 performs automated data triage to identify whether and how to feed back images (processed through the machine learning model to generate one or more inferences) into the learning workflow of the machine learning model. In some embodiments, the CPM assistance module 1000 also manages the machine learning models to control when to promote the models by determining model performance and automatically deploying models for use based on such performance, as appropriate. For example, as illustrated in FIG. 1A, the CPM assistance module 1000 may include data triage logic 1002 and, as appropriate, model promotion logic 1004. As used herein, the term "logic" may include an apparatus that performs a set of operations associated with the logic. For example, any of the logic elements included in the assistance module 1000 may be implemented by one or more computing devices programmed with instructions that cause one or more processing devices of the computing devices to perform a set of operations associated therewith. In certain embodiments, a logic element may include one or more non-transitory computer-readable media having instructions that, when executed by one or more processing devices of one or more computing devices, cause the one or more computing devices to perform an associated set of operations. As used herein, the term "module" may refer to a collection of one or more logic elements that together perform a function associated with the module. Different ones of the logic elements in a module may take the same form or different forms. For example, some logic in a module may be implemented by a programmed general-purpose processing device, while other logic in the module may be implemented by an application specific integrated circuit (ASIC). In another example, different ones of the logic elements in a module may be associated with different sets of instructions that are executed by one or more processing devices.A module may not include all of the logic elements shown in an associated drawing; for example, a module may include a subset of the logic elements shown in an associated drawing when that module performs a subset of the operations discussed in this specification with reference to that module.
[0020] The data triage logic 1002 can perform any of the data triage operations discussed herein. For example, the data triage logic 1002 can automatically identify data useful to a machine learning workflow and appropriately incorporate the identified data into the machine learning workflow. As described above, data triage reduces the overhead required for the learning workflow by reducing the number of required user annotations and other manual processing steps. Also, as described in more detail below, the data triage logic 1002 integrates the learning workflow into the customer's workflow to allow the customer to access and control the learning workflow of its models without requiring the customer to share image data or inferences with other customers and without requiring the customer to have experience or expertise in machine learning processes. For example, it may be advantageous if the data triage logic 1002 can deploy model learning at the customer's site level (e.g., server owner or server operated on behalf of the customer), require a small amount of user overhead, present a clearly understood process (without requiring the customer to have experience or training in machine learning processes), and provide reliable results.
[0021] FIG. 1B is a block diagram of data triage logic 1002, according to some embodiments. As illustrated in FIG. 1B, in some embodiments, data triage logic 1002 includes feature identification logic 1006, image selection logic 1008, training logic 1010, and user interface logic 1012. As described above, logic implemented via CPM assistance module 1000 (including data triage logic 1002) may be included in a single computing device or may be distributed across multiple computing devices in communication with each other as needed. For example, in some embodiments, feature identification logic 1006 may be executed via one or more computing devices included in or near a scientific instrument, such as, for example, an inference computer included in or near a CPM. Image selection logic 1008, training logic 1010, and user interface logic 1012 may be executed via computing devices remote from the CPM, such as a server in communication with one or more CPMs via one or more communication networks. For example, in some embodiments, the CPM may apply one or more machine learning models (e.g., via a local inference computer) to generate image data and generate inferences as described herein for feature identification logic 1006, and the image data and inferences are transmitted to one or more servers that apply image selection logic 1008, training logic 1010, and user interface logic 1012. In this sample configuration, user interface logic 1012 may generate a user interface that is provided to one or more user local computing devices. Further details regarding computing device configurations applicable to CPM assistance module 1000, and its associated logic, are provided below with respect to FIGS. 18 and 19.
[0022] The feature identification logic 1006 may use the machine learning model to generate one or more identified features in a set of images, such as a set of images generated via CPM. As described above, the one or more identified features may include stage detection, line indication end execution, device line termination, gliderator, image noise removal, or similar image features or artifacts.
[0023] The image selection logic 1008 determines whether the set of images meets one or more selection criteria that control whether or how the set of images is incorporated into the learning workflow of the machine learning model. In some embodiments, the image selection logic 1008 may determine whether the set of images meets one or more selection criteria by generating a set of metrics for one or more identified features associated with the set of images, where the set of images meets one or more selection criteria in response to one or more metrics included in the set of metrics meeting one or more predefined thresholds (also referred to as criteria). In addition to or as an alternative to using metrics for individual images or individual sets of images (e.g., related to the same sample or imaging session), the image selection logic 1008 may examine patterns or correlations between multiple sets of images to determine whether the set of images (or a portion thereof) meets one or more selection criteria. Identifying correlations or patterns, such as, for example, changes or trends in metrics for sets of images over time, may identify changes in conditions that may warrant new training of the machine learning model. Also, in some embodiments, the selection criteria is associated with random selection, such as, for example, selecting every 100th generated set of images and associated identified features for inclusion in the learning workflow. In addition to selecting which images to include in the learning workflow, the image selection logic 1008 can designate or flag images that include associated inferences (i.e., one or more identified features generated via a machine learning model) for inclusion in one or more different training datasets in the learning workflow. As used herein, a "training dataset" includes a set of images used as part of the machine learning workflow described herein, and as described herein, in some embodiments, the workflow uses multiple different training datasets.For example, in some embodiments, the training dataset includes a retraining dataset, an annotation dataset, a test dataset, and a validation dataset, and the image selection logic 1008 can automatically assign (e.g., designate or flag) one or more images and their associated inferences for inclusion in one or more of these training datasets. However, it should be understood that the embodiments described herein can use fewer or additional datasets than those described herein, or different types of datasets. In the example types of training datasets described herein, the retraining dataset can include images and associated inferences used to retrain the model, and the test dataset and validation dataset are used to test and validate the retrained model, respectively. The annotation dataset is stored, and images included in the annotation dataset may be available for manual review (e.g., by a CPM operator, a process engineer, or other user), and one or more user interfaces are provided that enable a user to review (using one or more visualization and navigation tools) an image or set of images and associated inferences, annotate one or more images as needed, include or exclude images from a learning process, specify a training dataset for one or more images in a learning workflow, or combinations thereof. In some embodiments, the image selection logic 1008 initiates one or more alerts (e.g., electronic messages such as email messages, text messages, chat messages, etc.) when one or more images are available for manual review in the annotation dataset. The alerts may include one or more selectable links (e.g., selecting a Uniform Resource Locator (URL)) to access one or more user interfaces (described below) that provide access to one or more images included in the annotation dataset for manual review.
[0024] The selection criteria may be based on one or more identified feature metrics (e.g., anomalies detected in an image or set of images compared to a base or criteria), patterns or correlations across a set of images, random selection, or a combination thereof. Also, in some embodiments, the selection criteria may be customized for a particular user or user site through one or more user-defined rules, which may be defined via one or more user interfaces provided via CPM assistance module 1000.
[0025] The training logic 1010 applies one or more of the training datasets established via the data triage performed in the feature identification logic 1006 and the image selection logic 1008 to train the machine learning model. In some embodiments, the training logic 1010 controls when training is performed. The training logic 1010 can trigger training of the model based on various conditions, such as, for example, the availability of annotations that correct inference errors included in the training dataset (e.g., determined based on the similarity between the inferences and the annotations), a predetermined increase (e.g., percentage) in the size of the training dataset, a predetermined increase (e.g., percentage) in annotations for a particular feature, availability of training resources, or based on a manually launched training job.
[0026] In response to determining to train the model with one or more available training datasets, the training logic 1010 may perform training according to a training configuration. Features of the training configuration may include one or more of: which algorithm to train, an initial transfer learning model to use, a training source (e.g., hardware selection and amount of parallelism (batch size, ROI size, GPU and nodes)), a training stopping condition (e.g., number of training epochs, convergence rate, lack of convergence, etc.), or a combination thereof. All or some features of the training configuration may be manually defined by a user through one or more user interfaces provided via the CPM assistance module 1000.
[0027] The user interface logic 1012 generates one or more user interfaces related to the functions performed via the data triage logic 1002. As described in more detail below, the one or more user interfaces may provide visualization, annotation, selection, and designation tools for reviewing image data included in the annotation dataset by the image selection logic 1008. Such user interfaces may enable a user to review images, review inferences associated with the images, add annotations, exclude images from a learning workflow, include images in a learning workflow, designate images for inclusion in a particular training dataset of a learning workflow, or combinations thereof. As described below with respect to FIG. 17, these user interfaces may display image data in a data display area 3002, display inferences associated with the image data, metrics associated with such inferences, or combinations thereof in a data analysis area 3004, and display options for annotating, excluding, including, or designating image data in a scientific instrument control area 3006, a settings area 3008, or combinations thereof.
[0028] Also, in some embodiments, the user interface logic 1012 generates one or more user interfaces that present options for configuring (setting and / or modifying) the data triage performed via the data triage logic 1002. For example, the user interface logic 1012 may generate one or more user interfaces for configuring selection criteria applied by the image selection logic 1008, configuring training trigger conditions applied by the training logic 1010, configuring training configurations applied by the training logic 1010, or a combination thereof.
[0029] The user interface generated via the user interface logic 1012 may include various access permissions that allow or restrict user interaction with data or options included in the user interface. For example, in some embodiments, reviewing images included in the annotation dataset, annotating images included in the annotation dataset, configuring selection criteria, configuring training aspects, etc. may be permitted only for users with specific access permissions. Such access permissions may be implemented to control access to data (e.g., only users associated with a particular customer may view image data collected via equipment associated with that customer), as well as to control which users may configure the CPM assistance module 1000 and its associated functionality. In some embodiments, the user interface logic 1012 may be distributed among multiple logic modules (e.g., first user interface logic, second user interface logic, etc.), each of which may generate and provide one or more specific user interfaces (e.g., a specific user interface for a particular output, input option, access permission, etc.).
[0030] As described above with respect to FIG. 1A, in some embodiments, the CPM assistance module 1000 also includes model promotion logic 1004. FIG. 1C is a block diagram of the model promotion logic 1004, according to some embodiments. The model promotion logic 1004 can perform any of the contemplated model promotion actions. As illustrated in FIG. 1C, in some embodiments, the model promotion logic 1004 includes model performance logic 1014, user interface logic 1016, and promotion logic 1018.
[0031] The model performance logic 1015 generates one or more performance metrics for the machine learning model. The performance metrics may be based on the training or testing performance of the model.
[0032] The promotion logic 1018 deploys a particular machine learning model to one or more scientific instruments, such as one or more CPMs, based on the performance metrics of the machine learning model. The promotion logic 1018 can compare performance metrics between models and rank candidate models based on test results, deployment performance, or a combination thereof. If desired, the promotion logic 1018 can be manually monitored. For example, through the promotion logic 1018, a user can enable a particular model to be deployed (e.g., for process stability or to screen new models). In the case of automatic deployment, the promotion logic can select the best model for a particular step of the process of interest and record which model was deployed where to ensure a clear record of the model deployment. In some embodiments, when automatic promotion, including automatic deployment, is used, the level of automation can be set by the user. Automatic promotion, including automatic deployment, reduces the time constraints often associated with model deployment as well as the associated effort and expertise to perform such deployment. Also, the ML workflow described herein can be used to create well-performing models that incrementally improve, with automatic promotion ensuring that such improvements are deployed appropriately and that newer, better performing models are used when they become available. Additionally, the promotion process and the ability for users to configure the process and see the results of the process (e.g., performance metrics, inference, rank, deployment, etc.) provide an observable process where the running model is known, the process is clear, the results are understandable, and the inference is clear.
[0033] The promotion logic 1018 can perform model promotion in one or more user-customizable promotion steps, such as, for example, "prototype," "certified," "production," etc. Each step can be associated with a particular threshold score, and the model's score can be based on model loss (tracked during model training), model testing (tracked during model testing, or validation), and model deployment (tracked during model use, such as by based on the metrics described above for inferences generated by the deployed model). In some embodiments, components of the score can be weighted differently within a step, weighted differently between steps, or a combination thereof.
[0034] For example, the promotion logic 1018 can apply promotion criteria to identify when to promote a model to a particular step. The promotion criteria can include a weighted score as described above. The promotion criteria can include other parameters such as training set size, training time or frequency. The promotion criteria can be set by a user via one or more user interfaces, which can be generated via the user interface logic 1016. The user interface logic 1016 can also generate one or more user interfaces that allow a user to manually promote (including deploy) a particular model to one or more devices. The user interface logic 106 can also generate one or more interfaces for reviewing or troubleshooting the performance of a model, such as by providing inferences generated by the model for particular image data (e.g., using various visualization and navigation tools). As described above with respect to the user interface logic 1012, the user interfaces generated via the user interface logic 1016 can include various access permissions that can allow or limit user interaction with data or options included in the user interface. For example, in some embodiments, only users with certain access permissions may be permitted to set promotion criteria, review or troubleshoot model performance, manually promote (including deploy) models, etc. Again, such access permissions may be implemented to control access to data (e.g., only users associated with a particular customer may view image data collected via devices associated with that customer), as well as to control which users may configure the CPM assistance module 1000 and its associated functionality.In some embodiments, the user interface logic 1016 may be distributed among multiple logic modules (e.g., first user interface logic, second user interface logic, etc.), each logic module may generate and provide one or more specific user interfaces (e.g., a specific user interface for a particular output, input option, access permission, etc.).
[0035] 2A is a flow diagram depicting a method 2000 performed by the CPM assistance module 1000. Although the operations of the method 2000 may be illustrated with reference to certain embodiments disclosed herein (e.g., the CPM assistance module 1000, or logic included therein, the graphical user interface 3000, the computing device 4000, and / or the scientific instrument assistance system 5000), the method 2000 may be used in any suitable configuration to perform any suitable assistance operations. Although each block of the method 2000 is illustrated in FIG. 2A once each and in a particular order, the operations may be appropriately reordered and / or repeated as desired (e.g., different operations performed may be performed in parallel, as appropriate).
[0036] As described below, method 2000 represents a method of assisting a scientific instrument, such as a CPM, as well as a machine learning model applied by such an instrument. Method 2000 is described herein with respect to a CPM. However, as noted above, method 200 may be used with other types of scientific instruments, including other types of microscopes or imaging instruments. It should also be understood that prior to performing method 2000, the scientific instrument (e.g., a CPM) may be appropriately prepared and configured for operation. For example, a sample may be selected and placed in a holder within a chamber of the CPM. The CPM (or associated computing device) may also be loaded with a machine learning model configured to generate one or more inferences (i.e., identified features) within an image generated by the CPM of the sample.
[0037] As illustrated in FIG. 2A , method 2000 includes performing a data triage operation (e.g., via data triage logic 1002 at block 2002) and, optionally, performing a model promotion operation (e.g., via model promotion logic 1004 at block 2004). Method 2000 may include some or all of the operations described with reference to FIG. 2A . For example, method 2000 may include performing both a data triage operation (block 2002) and a model promotion operation (block 2004). However, in other embodiments, method 2000 may only include performing a data triage operation (block 2004) and not a model promotion operation (block 2004). Similarly, in some embodiments, method 2000 may only include performing a model promotion operation (block 2004) and not a data triage operation (block 2002). Also, in some embodiments, the data triage operation (block 2002) may be performed before, in parallel with, or after the model promotion operation (block 2002). Moreover, method 2000, or portions thereof, may be repeated (as individual operations or as a series of operations). For example, the data triage operation (block 2002) may be repeated one or more times (e.g., to create "continuous" learning) concurrently with or separate from the performance of the model promotion operation (block 2004), which may be repeated as the model is promoted to various steps (e.g., "prototype," "certified," "production," etc.).
[0038] Figure 2B is a flow diagram illustrating the data triage operations performed in block 2002 of method 2000. As discussed above with respect to Figure 2A, each block of the flow diagram illustrated in Figure 2B is illustrated once and in a particular order, however, operations may be reordered and / or repeated as desired and appropriate (e.g., different operations performed may be performed in parallel, if preferred).
[0039] As seen in FIG. 2B, in block 2006, the feature identification logic 1006 uses the machine learning model to generate one or more identified features in a set of images, such as a set of images generated via a CPM or other scientific instrument. FIG. 3 illustrates an example image 2007 included in a set of images generated via a scientific instrument, and illustrates a number of features (individually referred to as “features 2008” or “identified features 2008” and collectively referred to as “features 2008” or “identified features 2008”) identified via a machine learning model applied to the image 2007. As illustrated in FIG. 3, in this example, the identified features 2008 represent line indication end (LIT) features identified in the image 2007, and in particular, six LIT features identified in the image 2007 via the machine learning model. In some embodiments, the user interface logic 1012 is configured to generate one or more user interfaces that display the image dataset and the identified features. For example, in some embodiments, the user interface logic 1012 generates a user interface that allows a user to scroll (e.g., using a slider or similar selection mechanism, gestures, commands, etc.) through a set of selected images, and for each displayed image of the set of images, one or more identified features are displayed within the image (e.g., as annotations, as illustrated in FIG. 3). The user interface may also be configured to allow a user to select a particular set of images, select particular features identified within the set of images, and combine these to display the selected images and the corresponding selected features.
[0040] Returning to FIG. 2B , at block 2009, the image selection logic 1008 determines whether the identified features determined at block 2006 satisfy one or more selection criteria. In some embodiments, the image selection criteria are based on one or more metrics associated with the identified features. The one or more metrics may be compared to one or more predefined thresholds representing expected metrics for the identified features to detect anomalies in the identified features that may indicate that the machine learning inference has failed or that they should not be included in a training dataset used for further training of the machine learning model. In some embodiments, comparing the metric to an associated predefined threshold includes comparing the metric to an expected value or expected range, determining the variance between the metric and the expected value (e.g., a threshold or a criterion) and comparing the variance to the expected value or expected range, or a combination thereof.
[0041] For example, one or more characteristics of the identified features in each image of the set of images may be plotted across the set of images, and the slope of this plot may be used as a metric for the identified features and, therefore, the set of images. In an embodiment in which the set of images are LIT runs and the identifier features are LIT features, the plotted characteristics may include the number of LIT features identified in each image, the area of the features identified in each image, or the distance of the features identified in each image. For example, FIG. 4 is an example plot 2010 showing the number of features identified per image of the set of images, FIG. 5 is an example plot 2012 showing the feature areas identified per image of the set of images, and FIG. 6 is an example plot 2014 showing the feature distances identified per image of the set of images. Using one of these example plots, the one or more metrics may include a slope mean, a slope standard deviation, a position change, or a combination thereof. The position change is a metric for the number of features, such as the number of features identified for a LIT run.
[0042] For example, the following equations can be used to calculate the slope mean, slope standard deviation, and position change associated with a plot of the number of features identified in each image of a set of images (see, e.g., FIG. 4): r = stats.linregress(x,y) slope.append(r.slope) runData['slope avg']=np.mean(np.absolute(slope)) runData['slope std']=np.std(np.absolute(slope)) runData['location change']=np.sum(np.absolute(np.gradient(runData['features']))).tolist()
[0043] A value calculated from the plot as described above may be compared to a threshold (e.g., an expected value) to determine whether the set of images meets one or more selection criteria. For example, under ideal conditions, the slope standard deviation of a plot of the number of features identified in a LIT run should be zero (indicating that the same number of features were identified in each image in the set of images). Thus, a large slope standard deviation in such a plot may indicate that LIT features were not properly identified by the machine learning model (i.e., machine learning inference failed). As another example, if six LIT features were expected in each image (e.g., based on known patterns on the samples), a value other than six for the position change metric associated with the number of identified features would indicate that LIT features were not properly identified via the machine learning model (i.e., machine learning inference failed).
[0044] For example, plot 2016 illustrated in FIG. 7 shows a plot of the number of features detected within an LIT run where the number of identified features varies between 4 and 7. Similarly, plot 2018 illustrated in FIG. 8 shows a plot of feature areas of features detected during an LIT run where the feature areas vary across the image, where it is expected that if all LIT features are properly identified, the detected area should remain approximately constant (e.g., with some variation as the boundaries of the LIT features are obscured by the changing sample features). Similarly, plot 2020 illustrated in FIG. 9 shows a plot of feature distances of features detected during an LIT run, where the feature distances are generally nonlinear, with a straight line distance being expected for properly identified LIT features. Thus, plots 2016, 2018, and 2020 each represent failed machine learning inferences that are not good candidates for use in further training of the machine learning model.
[0045] As described above, the image selection logic 1008 may compare the determined metric to one or more thresholds to determine whether a set of images with associated inferences should be automatically included in a particular training dataset of the machine learning model (block 2026) or automatically excluded from the training dataset (block 2024). In some embodiments, the image selection logic 1008 may be configured to apply different selection criteria (e.g., thresholds) to various training datasets. For example, if the determined metric does not meet a threshold associated with the training dataset (e.g., if the metric exceeds the threshold), the identified features may not represent inferences that would improve the performance of the machine learning model if fed back to the machine learning model as training data (e.g., the identified features do not accurately represent all of the LIT features that the model is supposed to identify). Thus, in this situation, if the metric does not meet a threshold associated with the training dataset (block 2022), the image selection logic 1008 may flag the image dataset as being excluded from the training dataset of the machine learning model (block 2024). Alternatively, if the metric meets a threshold associated with a training dataset (block 2022), the image selection logic 1008 may flag the set of images as being included in the training dataset (block 2026). As described above, the image selection logic 1008 may repeat this process (blocks 2022 and 2024 or 2026) for each of the training datasets associated with training the model (e.g., a retraining dataset, a testing dataset, a validation dataset, an annotation dataset, or a subset thereof). Alternatively, or additionally, the image selection logic 1008 may be configured to use one threshold for multiple training datasets.For example, in response to the metric not meeting a threshold for the retraining dataset, the image selection logic 1008 may be configured to flag the set of images for inclusion in the annotation dataset of the machine learning model, and a user can manually review the set of images and, if necessary, annotate (e.g., mark image features that were not identified by the machine learning model) and flag the set of (annotated) images for inclusion in the retraining dataset.
[0046] In some embodiments, the plots described above may be provided via one or more user interfaces generated via the user interface logic 1012, and in some embodiments, the one or more plots may be displayed along with the corresponding images and the identified features as described above with respect to block 2006. In some embodiments, the metrics determined by the image selection logic 1008 are also used to detect errors in an image run, such as an LIT run. For example, if the beam is out of focus on a patterned wafer, features may not be correctly identified via a machine learning model based on the quality of the image. Thus, based on the set of calculated metrics, the image selection logic 1008 may indicate an error, such as in a user interface, output or log the error, and alert a user that the run needs to be run again.
[0047] As described above, the one or more selection criteria may include one or more thresholds (e.g., representing expected metric values, or predetermined criteria) that the image selection logic 1008 compares to one or more metrics. That is, the one or more selection criteria may be used to identify anomalies in a particular image or set of images, and the metric may be used to automatically exclude a set of images from a retraining dataset or to automatically include a set of images in an annotation dataset. For example, the one or more selection criteria may include a predetermined criterion regarding a characteristic of the one or more identified features, and the image selection logic determines whether the set of images meets the one or more selection criteria by comparing the predetermined criterion to identify anomalies in the one or more identified features. The predetermined criterion regarding the characteristic of the one or more identified features may include at least one selected from the group consisting of a predetermined reference size of the one or more identified features, a predetermined reference number of the one or more identified features, a predetermined reference position of the one or more identified features, a predetermined reference shape of the one or more identified features, and a predetermined reference distance between two of the one or more identified features. The image selection logic may determine whether the set of images meets one or more selection criteria by comparing characteristics of the one or more identified features in a single image of the set of images to the predetermined criteria, or by comparing representative characteristics of the one or more identified features in multiple images included in the set of images to the predetermined criteria. The representative characteristics may include at least one selected from the group consisting of a mean value of the characteristic in the multiple images, an average of the characteristic in the multiple images, a median of the characteristic in the multiple images, a standard deviation of the characteristic in the multiple images, and a slope of a plot of the characteristic in the multiple images. In some embodiments, the predetermined criteria are user-defined and may be set based on one or more inputs or instructions received through one or more user interfaces.
[0048] Alternatively or additionally, the one or more selection criteria can compare metrics across the sets of images to identify patterns. For example, if a particular metric begins to vary across the sets of images, the image selection logic 1008 can be configured to automatically exclude older sets of images and automatically include more recently generated sets of images in a retraining dataset so that the machine learning model can be retrained on the current conditions or operational parameters of the scientific instrument, or to include one or more of the sets of images in the annotation dataset. Similarly, if a metric associated with a particular set of images differs from other sets of images generated by the scientific instrument by more than a predetermined amount, the selection criteria can dictate that the different set of images be included in the annotation dataset (e.g., regardless of whether the metric meets a threshold associated with the set of images). Thus, in some embodiments, the one or more selection criteria include characteristics of one or more identified features, and the image selection logic determines whether a set of images meets the one or more selection criteria by identifying a pattern in the characteristics across the sets of images, e.g., the change in the characteristics across the sets of images or the change in the characteristics across the sets of images exceeds a predetermined threshold, which may be a user-defined threshold. As described above with respect to detecting anomalies, the characteristics of the one or more identified features used in identifying the pattern may include at least one selected from the group consisting of a size of the one or more identified features, a number of the one or more identified features, a position of the one or more identified features, a shape of the one or more identified features, and a distance between two of the one or more identified features.
[0049] Alternatively, or additionally, the one or more selection criteria may include image quality parameters of the image or set of images, for example, the one or more selection criteria may exclude a set of images from the training dataset in response to the set of images including double images, out-of-focus or blurry portions, or other image artifacts.
[0050] Alternatively or additionally, the one or more selection criteria may include one or more random selection criteria. For example, the selection criteria may define that every 100 sets of generated images are included in the retraining dataset, the test dataset, the validation dataset, the annotation dataset, or a combination thereof. In some embodiments, different datasets may have different random selection criteria, for example, every 100 sets of images are included in the annotation dataset and every 50 sets of images are included in the retraining dataset. Thus, in some embodiments, the one or more selection criteria include random selection, which may define a predetermined frequency for including a set of images in the training dataset. As with other selection criteria, the random selection may be user-defined.
[0051] Any of the above-mentioned selection criteria may be established automatically (e.g., based on patterns or trends, such as based on multiple sets of images processed via a machine learning model) or may be manually defined by a user, such as, for example, through one or more user interfaces generated via the user interface logic 1012. For example, the one or more selection criteria may include user-defined rules based on one or more characteristics of the identified features as described above and various predefined thresholds or references. For example, the image selection logic 1008 may be configured to receive a first indication of one or more first selection criteria (e.g., through one or more user interfaces) and determine whether the first set of images meet the first selection criteria, and in response to a determination that the first set of images meets the first selection criteria, the first set of images is included in at least one of the datasets. The image selection logic 1008 may also be configured to receive (e.g., through one or more user interfaces) a second indication of one or more second selection criteria, the second selection criteria being different from the first selection criteria, and to determine whether the second set of images meet the second selection criteria, and in response to a determination that the second set of images meets the second selection criteria, the second set of images is included in at least one of the datasets. In some embodiments, the user interface provided (e.g., via the user interface logic 1012) may include a list of criteria available for selection by the user. For example, FIG. 10 shows a user interface 2028 that includes a list 2030 of available selection criteria (e.g., a list of different types of selection criteria). As illustrated in FIG. 10, the list 2030 includes an "anomaly" selection criterion type that, when selected, allows a user to configure rules for selecting an image or set of images to be included in the training dataset based on detected anomalies in the image or set of images, as described above.Similarly, the "slope", "locationchange", "area", "stderr", and "expected features" selection criteria types allow a user to configure rules for selecting an image or set of images based on one or more metrics or features detected in the image or set of images. Similarly, as illustrated in FIG. 10, list 2030 includes a "confidence" selection criteria type that, if selected by the user, allows the user to set a minimum confidence or probability level that will be applied when a decision is made to add a particular image or set of images to a particular training data set. For example, through user interface 2028, the user can set a confidence level of 75%, and if a decision by assistance module 1000 is associated with a confidence level that meets the minimum confidence level set by the user, the image or set of images will be assigned to the particular training data set.
[0052] In some embodiments, depending on the particular type of selection criterion selected by the user from the list 2030, the user interface 2028 provides one or more input or selection mechanisms for defining one or more details of the selection criterion. For example, as illustrated in FIG. 10, in response to receiving a selection of the “confidence” selection criterion from the list 2030, the user interface 2028 provides an author field 2032, a description field 2034, and a template field 2036. The author field 2032 allows the user to enter the “confidence” criterion or the name of the author of the rule. Also, in some embodiments, the author field 2032 may be automatically populated by the assistance module 1000 based on the user's login or other credentials. The description field 2034 allows the user to add a description or comment about the rule, and the template field 2036 allows the user to specify or select a stored template to represent the rule. For example, in some embodiments, the selection criteria (e.g., configured in the user interface 2028, other user interfaces, or in other ways) may be stored and reused.
[0053] After configuring any desired selection criteria through the user interface 2028, the user can select a "Launch" selection mechanism 2040 to schedule the triage workflow (e.g., evaluate acquired images according to the configured selection criteria) or at least access the next user interface or step in the triage workflow configuration process. The user can select a "Clear Output" selection mechanism 2042 to clear the inputs presented in the user interface 2028, such as details of a particular type of selection mechanism.
[0054] In some embodiments, the one or more selection criteria define a set of images to be included in each dataset. However, in other embodiments, the selection criteria may define a set of images to be included in a subset of the dataset. For example, in some embodiments, the one or more selection criteria define a set of images to be included in the retraining dataset and the annotation dataset. The image selection logic 1008 may be configured to distribute the images included in the retraining dataset to the test dataset, the validation dataset, or both. This distribution may be performed randomly (e.g., according to a predetermined ratio: 50% for training, 25% for testing, 25% for validation) or based on a metric or parameter of the images included in the retraining dataset.
[0055] As described above, the images included in the annotation dataset are accessible through one or more user interfaces, and a user may review the images, manually include the images in the selected dataset, exclude the images from the selected dataset, add annotations to the images (e.g., correct anomalies detected in the features identified via the machine learning model), or combinations thereof. For example, the feature identification logic 1006 may generate, via the machine learning model, one or more first identified features in a first set of images acquired via the scientific instrument, and generate, via the machine learning model, one or more second identified features in a second set of images acquired via the scientific instrument. The image selection logic 1008 (via the user interface 1012) may provide the first set of images and the one or more first identified features to the user interface, and provide the second set of images and the one or more second identified features to the user interface, such that in response to a first instruction by the user through the user interface, the first set of images is excluded from a training set for the machine learning model, and in response to a second instruction by the user through the user interface, the second set of images is included in a training set for the machine learning model. The user interface may provide one or more input mechanisms, selection mechanisms, or combinations thereof that allow a user to manually assign a particular image or set of images to a particular training data set. The input mechanisms or selection mechanisms may include a drop-down menu that allows a user to select an "assign to..." menu option, a button designated for a particular training data set that a user can select to manually assign an image or set of images to a specified training data set, a drop-and-drag function that allows a user to move images or sets of images within the user interface to manually assign the images or sets of images to a particular training data set, etc.
[0056] Thus, under the control of the automated machine learning workflow of the assistance module 1000, the user can access a list of images (e.g., LIT runs) prepared for review along with a corresponding set of candidate annotations (i.e., inferences generated via the machine learning model). Thus, rather than being tasked with reviewing all inferences, the above workflow creates a limited set of images (and corresponding inferences) for the user to manually review. As mentioned above, creating such a limited list is advantageous compared to traditional manual approaches that present the user with a seemingly endless list of images that must be worked through to identify anomalies or inferences that must be manually corrected or that must rely only on sporadic manual inference checking (increasing the likelihood of missing many relevant inference errors that, once corrected, would create valuable training data for the machine learning model). In some embodiments, the user can assign images that are initially included in the annotation dataset to a specific different dataset (e.g., a retraining dataset, a test dataset, or a validation dataset). In other embodiments, a user may indicate that an image should be included in a training dataset (e.g., without specifying a particular training dataset), and the image selection logic 1008 may be configured to automatically assign the included images to the appropriate dataset. A user may also flag a particular image as not being included in any training dataset used to train the machine learning model.
[0057] In some embodiments, the image selection logic 1008 generates and transmits at least one alert regarding the image and one or more associated identified features available via one or more user interfaces. The alert may be transmitted via at least one selected from the group consisting of an email, a text message, and a software notification.
[0058] 2B , at block 2040, the training logic 1010 retrains the machine learning model using the set of images and associated identified features (machine learning inferences) contained in one or more of the available datasets (e.g., the retraining dataset, the test dataset, and the validation dataset). For example, in some embodiments, the training logic 1010 retrains the machine learning model using the retraining dataset and tests and validates the (retrained) machine learning model using the test dataset and the validation dataset, respectively.
[0059] In some embodiments, the training logic 1010 retrains the machine learning model in response to a trigger event. The trigger event may be based on the number of user-annotated images included in the training set, an increase in the size of the training set, an increase in the number of user-annotated images (e.g., overall or for a given feature), the availability of one or more training resources, or manual initiation by a user (e.g., received through a user interface generated via the user interface logic 1012). Thus, as described above, once a set of images is generated via the scientific instrument and one or more identified features are generated via the machine learning model, the training logic 1010 retrains the machine learning model using the generated training dataset in response to the occurrence of a trigger event by automatically processing the set of images (including the identified features) as described above to identify which training dataset images (including associated identified features) should be included or excluded.
[0060] The training logic 1010 can perform training of the machine learning model according to a training configuration. The training configuration can include one or more training characteristics such as, but not limited to, a decision on which model to train, an initial transfer of the learned model to a training set, training resources to use (e.g., hardware selection, amount of parallelism, batch size, return on investment, available graphical processing units, nodes), training stopping conditions (e.g., threshold number of training epochs, convergence rate, lack of convergence), or a combination thereof.
[0061] In some embodiments, retraining a machine learning model can consume significant computing resources. To address this issue, in some embodiments, the training logic 1010 defines the training job within a workflow engine configured to manage parallel jobs, such as, for example, Argo workflows on Kubernetes, to ensure that training (and promotion, if necessary, as described below) can occur even when computing resources are scarce, by acquiring resources for the training job when they are needed and releasing them when the tasks are completed.
[0062] During retraining, the training losses of the machine learning models may be stored and compared, and may be used to determine model performance and promote the models appropriately, as described in more detail below.
[0063] Repeating the data triage operations illustrated in FIG. 2B can create a “continuous” learning workflow for the machine learning model. This “continuous” learning workflow establishes continuous monitoring and improvement of the machine learning model to improve not only the model’s qualification but also its performance through iterative training with customer-specific data. Improved machine learning model performance can lead to further improvements in the operation of scientific instruments and associated processes, e.g., more accurate sample preparation accuracy and image quality.
[0064] As described above, the user interface logic 1012 may be configured to provide one or more user interfaces associated with the automated data triage process. All or a portion of the user interfaces generated as part of performing the data triage operations may include similar features, components, and functionality as described below with respect to the graphical user interface 3000. For example, in addition to providing a user interface that allows a user to provide feedback regarding, inclusion and exclusion of, images and, where appropriate, annotations that are automatically included in a particular training data set, the user interface logic 1012 may provide one or more user interfaces that allow a user to review the images and associated inferences, set and modify one or more selection criteria used by the image selection logic 1008, request that data triage be rerun (e.g., after modifying one or more selection criteria), set or modify training configurations applied by the training logic 1010, or combinations thereof. Through the user interfaces, the user may also control the amount of automation applied via the assistance module 1000 during data triage. For example, in some embodiments, a user may configure the assistance module 1000 to perform data triage in a fully automated manner (e.g., without prompting the user to review images included in the annotation dataset). Also, model training results may be presented through one or more user interfaces. FIG. 11 shows an example user interface 2045 that provides training information. As illustrated in FIG. 11, the user interface 2045 may provide a plot 2047 showing the average training loss per epoch and the average validation loss per epoch. The user interface 2045 may also provide a test segment 2048, in which a left image 2048a in the test segment 2048 represents a ground truth image and a right image 2048b in the test segment 2048 represents an associated inference generated via the machine learning model.The user interface 2045 may also include a slider or other selection mechanism 2049 that allows the user to scroll through the test segments. In some embodiments, the user interface 2045 also includes one or more selection mechanisms that allow the user to select a particular train from a list of model trains.
[0065] 2B , in addition to performing data triage operations (block 2002), as discussed above, the assistance module 1000 may also be configured to perform model promotion operations (such as, for example, at block 2004, via the model promotion logic 1004). Again, as understood above, in some embodiments the assistance module 1000 may be configured to perform only data triage operations (block 2002) or only model promotion operations (block 2004), and in some embodiments the assistance module 1000 is configured to perform the data triage operations (block 2002), the model promotion operations (block 2004), or combinations thereof, repeatedly or in various orders or arrangements, including in parallel, serial, or combinations thereof.
[0066] As described in more detail below, the model promotion logic 1004 is configured to automatically score, select, and deploy machine learning models. Other approaches to deploying machine learning models generally rely on machine learning practitioners, to whom customer data is provided for use in generating and deploying machine learning models. These experts perform, view, and evaluate various steps of the machine learning workflow, test and deploy machine learning models to one or more customers, and such models are often frozen for months or years, relying on the experts to manage the improvement or retraining of the models. As previously discussed, images created by scientific instruments are often considered confidential and proprietary, and such users often do not want to share such images. Thus, the automated deployment of models performed by the model promotion logic 1004 improves microscopy technology by taking advantage of the training of machine learning algorithms on user data (e.g., including improved accuracy, robustness, and execution speed) without requiring the disclosure of confidential and proprietary data to the experts. On the other hand, the model promotion logic 1004 can be deployed on customer computing environments without the need for intervention and management by machine learning experts. Thus, the model promotion logic 1004 enables efficient push-out of changes (i.e., retrained models, or models that outperform other available models) to a scientific instrumentation fleet (e.g., CPM) based on customer-specific improvements. For example, the model promotion logic may be configured to test models and compare models to identify the model that best achieves an objective, and this "best" model may then be deployed (e.g., with or without human oversight).
[0067] FIG. 2C is a flow diagram representing a model promotion operation performed at block 2004 of method 2000 (e.g., via model performance logic 1014, user interface logic 1016, promotion logic 1018, or a combination thereof), according to some embodiments. As described above with respect to FIG. 2A, each block of the flow diagram illustrated in FIG. 2C is illustrated once each and in a particular order. However, the operations may be reordered and / or repeated as desired and appropriate (e.g., different operations performed may be performed in parallel, if suitable). For example, in some embodiments, the model promotion operation (block 2004) may be performed according to a predetermined schedule or frequency (e.g., once a week, once a month), in response to a trigger condition (e.g., after a machine learning model has been retrained, or deployed for a predetermined amount of time, or applied to a predetermined number of images), in response to a manual initiation, or in response to a combination thereof.
[0068] As illustrated in FIG. 2C , at block 2500, the model performance logic 1014 generates one or more performance measures for each of one or more machine learning models, e.g., each of a plurality of machine learning models associated with a particular customer, set of scientific equipment, etc.
[0069] The model performance logic 1014 can take into account various parameters of the machine learning model to generate the performance metrics. For example, as described above, training losses may be stored by the training logic 1010 as part of retraining the machine learning model, and in some embodiments, the model performance logic 1014 uses these losses to generate the performance metrics for the model.
[0070] Alternatively or additionally, the performance metric may be based on offline testing performed by the model performance logic 1014 to score the performance of the machine learning model. In some embodiments, a lower score indicates a better performing model. In other embodiments, a higher score indicates a better performing model. The performance metric may include segmentation accuracy and similarity, inference time, confusion, or one or more process-specific metrics. The process-specific metrics may be based on expected characteristics of inferences on specific samples, such as percent mode error, percent feature error, mean slope, slope standard deviation, mean standard error, or combinations thereof. The process-specific metrics are generated using a separate data set. The offline testing may be customized for a particular machine learning model and may include one or more sub-tests. Test results may also be combined into a single score for model comparison and promotion.
[0071] For example, the model performance logic 1014 may implement a LIT test that includes two test metrics, e.g., "linearity standard error," which indicates accuracy, and "feature change," which indicates robustness. By way of example, FIG. 12 shows a plot 2052 illustrating the linearity standard error metric, and FIG. 13 shows a plot 2054 illustrating the feature change metric. The model performance logic 1014 may evaluate candidate models against stored test data sets and combine the test results in an appropriate manner (e.g., as a weighted sum).
[0072] In some embodiments, the model performance logic 1014 can apply a common test data set across multiple models to compare performance between each of the multiple models. In some embodiments, updating of such a common test data set can trigger execution of model promotion operations (block 2004) as described herein.
[0073] In some embodiments, a set of test results for the model may be provided to a user via a graphical user interface 3000. Each set of test results may include a graphical representation of the model's performance metrics. For example, the model performance logic 1014 (through the user interface logic 1016) may provide a plot, table, identified features, or other graphical representation of the model's performance metrics. FIG. 14 shows a graph 2056 of performance metrics for multiple machine learning models that may be presented to a user in one or more user interfaces.
[0074] Returning to FIG. 2C, at block 2070, the promotion logic 1018 determines whether one or more performance metrics for the model meet the promotion criteria. The promotion criteria may be based on a comparison of the performance metrics of the various models, and in some embodiments, the various performance metrics may be weighted differently (e.g., based on the level of importance of the tests used to generate the performance metrics). The promotion logic 1018 may be configured to apply a default comparison algorithm, but may be configured to allow the user to override this algorithm or portions thereof. The default comparison algorithm may assign each performance metric to one or more categories (e.g., error, validity, etc.), normalize and scale each performance metric such that higher values indicate higher importance, sum the performance metrics for each category to create a score for the category, and create a weighted sum from each category sum (i.e., create a composite model score). In some embodiments, each category may define a limit that excludes infeasible outcomes, and the composite model score may represent a weighted sum of the feasible outcomes from each category. Similarly, in some embodiments, the composite model score may be used to classify all models as either feasible or infeasible. The composite model scores of all feasible models can then be compared to identify the "best," or highest performing, model. In some embodiments, one or more thresholds can also be applied to the composite model score to determine whether a model meets the promotion criteria.
[0075] In response to the performance metrics of the machine learning model meeting the promotion criteria, the promotion logic 1018 may automatically promote the model. Instead of auto-promoting the model, one or more user interfaces or alerts may be generated to notify the user and prompt the user to confirm the promotion when the model meets the promotion criteria. In some embodiments, the assistance module 1000 allows the user to configure the level of automatic or manual promotion, and in some embodiments, the promotion logic 1018 may apply a combination of automatic and manual promotion. For example, in some embodiments, the promotion logic 1018 may be configured to compare the highest composite model score among the multiple models to a threshold, and if the score meets the threshold, the promotion logic 1018 may promote the model. However, in response to the highest composite score not meeting the threshold, the promotion logic 1018 may prompt the user (e.g., through the user interface logic 1016) to confirm whether to promote any of the models. In some embodiments, the promotion logic 1018 also applies other conditions in determining whether to promote the model (e.g., the conditions the model was trained on, such as the features the model identifies, the types or sizes of images the model was trained on, etc.).
[0076] Promoting the model may include deploying the model for use by the scientific instrument in performing feature identification on the generated images (block 2080). However, in other embodiments, the promotion logic 1018 may be configured to promote the model through multiple steps or states, such as, for example, "prototype," "qualified," and "production." When multiple steps are used, the promotion logic 1018 may be configured to generate losses, test, and deploy scores for each step, which may be weighted to identify eligible models (e.g., using step-specific promotion criteria).
[0077] In some embodiments, the promotion logic 1018 can manage the promotion of models using a finite state machine ("FSM") where step transitions are based on a set of transition rules. For example, the FSM can include a "candidate" state, which corresponds to all trained models that have not transitioned to other steps. In some embodiments, the "candidate" state corresponds to all models that have been determined to be feasible. The FSM can then transition to a "certified" state, which corresponds to models that meet customizable rules, such as, for example, the size of the training set, sample types for a particular customer, number of valid runs across multiple tools by a particular process engineer, a validity score above a configurable threshold, and / or an error score below a threshold. To move from the "certified" state to the "production" state, rules may be met, such as, for example, a higher number of runs, one or more test thresholds, approval by a process engineer, or other criteria. For example, in some embodiments, a user may certify new models to the "production" state during the day shift when more process support is available, but limit the night shift to either fixed models or the highest scoring models in the "production" state. In some embodiments, a user can define, via the graphical user interface 3000, the levels (e.g., operating time, test results) for promoting a model between any of the available steps or states, including a “production” or deployed state, manual approval, eligibility per customer fab process, etc.
[0078] In some embodiments, to deploy the model to the scientific instrument, communication is established between the scientific instrument and the machine learning server (which stores the model) via appropriate elements included in the scientific instrument support system 5000. In some embodiments, if one or more elements of the scientific instrument support system 5000 are not included in the user's communication network, a scientific instrument cluster network may be deployed to establish communication. Once communication is established, the machine learning server may identify inference computers (i.e., computers configured to apply machine learning models to sets of images) associated with the scientific instrument and establish bidirectional communication by creating a directory of inference computers and downloading communication address credentials to each inference computer. To receive one or more new models, the scientific instrument may register one or more model deployment criteria with the machine learning server. The model deployment criteria may include, for example, a particular inference, model state, and / or a particular model instance desired to receive. FIG. 15 illustrates an example model deployment criteria 2090 associated with a particular scientific instrument for registration with the machine learning server.
[0079] After registration, the machine learning server provides one or more models to the registered scientific instrument that meet one or more model deployment criteria. In some embodiments, if a new model is promoted and meets one or more model deployment criteria, the model is automatically downloaded from the machine learning server to the scientific instrument and loaded into the inference computer. Once loaded, the next inference call associated with the scientific instrument may use the newly downloaded model.
[0080] As mentioned above, in some embodiments, the assistance module 1000 allows a user to manually control the deployment of models. For example, FIG. 16 illustrates a portion of a user interface 2095 displaying a list of selectable model sources 2096 and a list of selectable models 2097 available at a model source selected in the list of selectable model sources 2096. Similarly, the user interface 2095 includes a list of selectable scientific instruments 2098 and a list of models 2099 representing models that have been deployed to the scientific instruments selected in the list of selectable scientific instruments 2098. The user interface 2095 further includes a copy selection mechanism 2100A and a delete selection mechanism 2100B. In response to receiving a selection of the copy selection mechanism 2100A, the model selected in the list of selectable models 2097 is deployed to the instruments selected in the list of selectable scientific instruments 2098. In response to receiving a selection of the delete selection mechanism 2100B, the model selected in the list of models 2099 is removed from the scientific instruments selected in the list of selectable scientific instruments 2098 (i.e., will never be deployed or used again).
[0081] As mentioned above, the scientific instrument assistance methods disclosed herein may include interactions with a human user (e.g., via a user local computing device 5020 discussed herein with reference to FIG. 19). These interactions may include providing information to the user (e.g., information regarding the operation of the scientific instrument, such as the scientific instrument 5010 of FIG. 19, e.g., inferences generated via one or more machine learning models about a set of images generated via the scientific instrument, information about a sample being analyzed, or other tests or measurements performed by the scientific instrument, information obtained from a local or remote database or other data storage device or arrangement, or other information, etc.), or options for the user to enter commands (e.g., to control the operation of the scientific instrument, such as the scientific instrument 5010 of FIG. 19, or to control the analysis of data generated by the scientific instrument), queries (e.g., directed to a local or remote database or other data storage device or arrangement), or other information. In some embodiments, these interactions may be performed through a graphical user interface (GUI) that includes a visual display on a display device (e.g., display device 4010, discussed herein with reference to FIG. 18) that provides output to the user and / or prompts the user to provide input (e.g., via one or more input devices such as a keyboard, mouse, trackpad, or touch screen, included in other I / O devices 4012, discussed herein with reference to FIG. 18). The scientific instrument support systems disclosed herein may include any suitable GUI for interaction with a user.
[0082] 17 shows an example graphical user interface 3000 that may be used in implementing some or all of the assistance methods disclosed herein, according to various embodiments. As described above, the graphical user interface 3000 may be provided on a display device (e.g., the display device 4010 discussed herein with reference to FIG. 18) of a computing device (e.g., the computing device 4000 discussed herein with reference to FIG. 18) of a scientific instrument assistance system (e.g., the scientific instrument assistance system 5000 discussed herein with reference to FIG. 19), and a user may interact with the graphical user interface 3000 using any suitable input device (e.g., any of the input devices included in the other I / O devices 4012 discussed herein with reference to FIG. 18) and input technology (e.g., cursor movement, motion capture, face recognition, gesture detection, voice recognition, button activation, etc.).
[0083] The graphical user interface 3000 may include a data display area 3002, a data analysis area 3004, a scientific instrument control area 3006, and a settings area 3008. The particular number and arrangement of areas shown in Figure 17 are merely illustrative, and any number and arrangement of areas, including any desired characteristics, may be included in the graphical user interface 3000.
[0084] The data display area 3002 can display data generated by a scientific instrument (e.g., the scientific instrument 5010 discussed herein with reference to FIG. 19). For example, the data display area 3002 may display any suitable data generated during the performance of a data triage operation (block 2002), a model promotion operation (block 2004), or a combination thereof, such as images generated via the scientific instrument, identified features (i.e., inferences) generated by a machine learning model applied to the images, etc.
[0085] The data analysis area 3004 may display the results of the data analysis (e.g., the results of analyzing the data illustrated in the data display area 3002 and / or other data). For example, the data analysis area 3004 may display any suitable data generated during the execution of a data triage operation (block 2002), a model promotion operation (block 2004), or a combination thereof, such as inference metrics and plots representing the same, training results, performance metrics, etc. In some embodiments, the data display area 3002 and the data analysis area 3004 may be combined in the graphical user interface 3000 (e.g., to include data output from scientific instruments and any analysis related to the data in a common graph or area).
[0086] The scientific instrument control area 3006 may include options that enable a user to control the scientific instrument (e.g., the scientific instrument 5010 discussed herein with reference to FIG. 19). For example, the scientific instrument control area 3006 may include any suitable of the options or control functions provided during the performance of a data triage operation (block 2002), a model promotion operation (block 2004), or combinations thereof described above, such as options to set and modify image selection criteria, options to manually include or exclude images, options to annotate images, options to set or modify training configurations, options to set and modify promotion criteria, options to manually deploy models, etc.
[0087] The settings area 3008 may include options that enable a user to control features and functions of the graphical user interface 3000 (and / or other GUIs) and / or perform common computing operations related to the data display area 3002 and the data analysis area 3004 (e.g., saving data on a storage device such as storage device 4004 as discussed herein with reference to FIG. 18, sending data to another user, labeling data, etc.). For example, the settings area 3008 may include any suitable settings associated with performing a data triage operation (block 2002), a model promotion operation (block 2004), or a combination thereof, such as, for example, annotating images, manually including or excluding images, registering with a machine learning server, communicating model deployment criteria communications, etc.
[0088] As noted above, the assistance module 1000 may be implemented by one or more computing devices. FIG. 18 is a block diagram of a computing device 4000 capable of implementing some or all of the scientific instrument assistance methods disclosed herein, according to various embodiments. In some embodiments, the CPM assistance module 1000 may be implemented by a single computing device 4000 or multiple computing devices 4000. Additionally, as discussed below, a computing device 4000 (or multiple computing devices 4000) implementing the CPM assistance module 1000 may be part of one or more of the scientific instrument 5010, user local computing device 5020, service local computing device 5030, or remote computing device 5040 of FIG. 19.
[0089] Although the computing device 4000 of FIG. 18 is illustrated as having several components, any one or more of these components may be omitted or duplicated as appropriate for the application and setting. In some embodiments, some or all of the components included in the computing device 4000 may be mounted on one or more motherboards and encapsulated in a housing (e.g., comprising plastic, metal, and / or other materials). In some embodiments, some of these components may be fabricated on a single system-on-a-chip (SoC) (e.g., an SoC may include one or more processing devices 4002 and one or more storage devices 4004). Additionally, in various embodiments, computing device 4000 may not include one or more of the components illustrated in FIG. 18, but may include interface circuitry (not shown) for coupling to one or more components using any suitable interface (e.g., a Universal Serial Bus (USB) interface, a High-Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface). For example, computing device 4000 may not include display device 4010, but may include display device interface circuitry (e.g., connectors and driver circuitry) to allow display device 4010 to be coupled.
[0090] The computing device 4000 may include a processing device 4002 (e.g., one or more processing devices). As used herein, the term "processing device" may refer to any device or portion of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that may be stored in registers and / or memory. The processing device 4002 may include one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), cryptographic processors (dedicated processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices.
[0091] The computing device 4000 may include a storage device 4004 (e.g., one or more storage devices). The storage device 4004 may include one or more memory devices, such as random access memory (RAM) (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive-bridging RAM (CBRAM) devices), hard-drive-based memory devices, solid-state memory devices, network drives, cloud drives, or any combination of memory devices. In some embodiments, the storage device 4004 may include memory that shares a die with the processing device 4002. In such embodiments, the memory may be used as a cache memory and may include, for example, embedded dynamic random access memory (eDRAM), or spin transfer torque magnetic random access memory (STT-MRAM). In some embodiments, the storage device 4004 may include a non-transitory computer-readable medium having instructions that, when executed by one or more processing devices (e.g., the processing device 4002), cause the computing device 4000 to perform any suitable method, or portion thereof, of the methods disclosed herein.
[0092] The computing device 4000 may include an interface device 4006 (e.g., one or more interface devices 4006). The interface device 4006 may include one or more communication chips, connectors, and / or other hardware and software to manage communications between the computing device 4000 and other computing devices. For example, the interface device 4006 may include circuitry to manage wireless communications to transfer data to and from the computing device 4000. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that may communicate data through the use of modulated electromagnetic radiation over a non-solid medium. This term does not imply that the associated devices do not include any wiring, although in some embodiments they may not. The circuitry included in the interface device 4006 for managing wireless communications may implement any of a number of wireless standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Institute for Electrical and Electronic Engineers (IEEE) standards including the Long-Term Evolution (LTE) project (e.g., advanced LTE project, ultra mobile broadband (UMB) project (also referred to as "3GPP2"), etc.) with any amendments, updates, and / or revisions.In some embodiments, the circuitry included in the interface device 4006 for managing wireless communications may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. In some embodiments, the circuitry included in the interface device 4006 for managing wireless communications may operate in accordance with an Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, the circuitry included in the interface device 4006 for managing wireless communications may operate in accordance with a Code Division Multiple Access (CDMA) network. Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and their derivatives, as well as any other wireless protocols designated as 3G, 4G, 5G, and beyond. In some embodiments, the interface device 4006 may include one or more antennas (e.g., one or more antenna arrays) for receiving and / or transmitting wireless communications.
[0093] In some embodiments, the interface device 4006 may include circuitry for managing wired communications, such as electrical, optical, or any other suitable communication protocol. For example, the interface device 4006 may include circuitry supporting communications according to Ethernet technology. In some embodiments, the interface device 4006 may support both wireless and wired communications, and / or multiple wired and / or wireless communications protocols. For example, a first set of circuits of the interface device 4006 may be dedicated to short-range wireless communications, such as Wi-Fi or Bluetooth, and a second set of circuits of the interface device 4006 may be dedicated to long-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first set of circuits of the interface device 4006 may be dedicated to wireless communications, and the second set of circuits of the interface device 4006 may be dedicated to wired communications.
[0094] The computing device 4000 may include a battery / power circuit 4008. The battery / power circuit 4008 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 4000 to an energy source separate from the computing device 4000 (e.g., AC line power).
[0095] The computing device 4000 may include a display device 4010 (e.g., multiple display devices). The display device 4010 may include any visual indicator, such as a head-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light emitting diode display, or a flat panel display.
[0096] The computing device 4000 may include other input / output (I / O) devices 4012. The other I / O devices 4012 may include, for example, one or more audio output devices (e.g., a speaker, a headset, an earpiece, an alarm, etc.), one or more audio input devices (e.g., a microphone or microphone array), a location device (e.g., a GPS device that communicates with a satellite-based system to receive the location of the computing device 4000, as known in the art), an audio codec, a video codec, a printer, sensors (e.g., a thermocouple or other temperature sensor, a humidity sensor, a pressure sensor, a vibration sensor, an accelerometer, a gyroscope, etc.), an image capture device such as a camera, a cursor control device such as a keyboard, a mouse, a stylus, a trackball, or a touchpad, a barcode reader, a Quick Response (QR) code reader, or a radio frequency identification (RFID) reader.
[0097] The computing device 4000 may have any form factor suitable for its application and configuration, such as a handheld or mobile computing device (e.g., a mobile phone, a smartphone, a mobile Internet device, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultra-mobile personal computer, etc.), a desktop computing device, or a server computing device or other networked computing component).
[0098] One or more computing devices implementing any of the CPM assistance logic or methods disclosed herein may be part of a scientific instrument assistance system. Figure 19 is a block diagram of an example scientific instrument assistance system 5000 capable of implementing some or all of the scientific instrument assistance methods disclosed herein, according to various embodiments. The CPM assistance apparatus and methods disclosed herein (e.g., the CPM assistance module 1000 of Figures 1A, 1B, and 1C and the method 2000 of Figures 2A, 2B, and 2C) may be implemented by one or more of the scientific instruments 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 of the scientific instrument assistance system 5000.
[0099] Any of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may include any of the embodiments of the computing device 4000 discussed herein with reference to FIG. 18, and any of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may take the form of any suitable embodiment of the computing device 4000 discussed herein with reference to FIG. 18.
[0100] Each of the scientific instruments 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may include a processing device 5002, a storage device 5004, and an interface device 5006. The processing device 5002 may take any suitable form, including any form of the processing device 4002 discussed herein with reference to Figure 18, and the processing devices 5002 included in different ones of the scientific instruments 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may take the same form or different forms. The storage device 5004 may take any suitable form, including any form of the storage device 4004 discussed herein with reference to Figure 18, and the storage devices 5004 included in different ones of the scientific instruments 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may take the same form or different forms. The interface device 5006 may take any suitable form, including any of the forms of the interface devices 4006 discussed herein with reference to FIG. 18, and the interface devices 5006 included in different ones of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may take the same or different forms.
[0101] The scientific instruments 5010, the user local computing device 5020, the service local computing device 5030, and the remote computing device 5040 may communicate with other elements of the scientific instrument support system 5000 via communication paths 5008. As shown, the communication paths 5008 may be communicatively coupled to interface devices 5006 of the various elements of the scientific instrument support system 5000, and may be wired or wireless communication paths (e.g., in accordance with any of the communication techniques discussed herein, with reference to the interface device 4006 of the computing device 4000 of FIG. 18). While the particular scientific instrument support system 5000 shown in FIG. 19 includes communication paths between each pair of the scientific instruments 5010, the user local computing device 5020, the service local computing device 5030, and the remote computing device 5040, this "fully connected" implementation is merely exemplary, and in various embodiments, various ones of the communication paths 5008 may not be present. For example, in some embodiments, the service local computing device 5030 may not have a direct communication path 5008 between its interface device 5006 and the interface device 5006 of the scientific instrument 5010, and instead may communicate with the scientific instrument 5010 via a communication path 5008 between the service local computing device 5030 and the user local computing device 5020, and a communication path 5008 between the user local computing device 5020 and the scientific instrument 5010.
[0102] In some embodiments, the scientific instrument 5010 includes any suitable CPM, such as a scanning electron microscope (SEM), a transmission electron microscope (TEM), a scanning transmission electron microscope (STEM), or an ion beam microscope (and may include other scientific instruments). For example, FIG. 20 illustrates a scientific instrument 5010 implemented as a CPM 6000, according to some embodiments. The CPM 6000 illustrated in FIG. 20 represents a scanning electron microscopy with energy dispersive X-ray spectroscopy (SEM / EDX) system. However, as previously mentioned, the CPM 6000 illustrated in FIG. 20 is provided as one example type of CPM, and the support methods described herein may be used with other types of CPMs or with other types of scientific instruments. As illustrated in FIG. 20, the CPM 6000 includes a particle-optical column 6015 attached to a vacuum chamber 6006. Within the particle-optical column 6015, electrons generated by an electron source 6012 are modified by a compound lens system 6014 and then focused by a lens system 6016 onto the sample 6002 as an incident beam 6004. The incident beam 6004 may be scanned across the sample 6002 by manipulating a scanning coil 6013. The sample may be held by a sample stage 6008. The CPM 6000 may include multiple detectors to detect various emissions from the sample 6002 in response to illumination with the incident beam 6004. The first detector 6003 may detect X-rays emitted from the sample 6002. In one example, the detector 6003 may be a multi-channel photon counting EDX detector. The second detector 6001 may detect electrons, such as backscattered and / or secondary electrons, emitted from the sample 6002. In one example, the detector 6001 may be a segmented electron detector. As illustrated in Figure 9, CPM 6000 also includes a computing device 4000, as generally described above with respect to Figure 18. Computing device 4000 may be configured to transmit and receive one or more control signals as described below, and in some embodiments, can perform the assistance methods described herein.For example, computing device 4000 may be configured to perform data triage operations (block 2002), model promotion operations (block 2004), or a combination or subset thereof. For example, in some embodiments, computing device 4000 may be configured to generate a set of images and one or more identified features, and thus may be referred to as an "inference" computer or computing device. As noted above, the set of images and the associated one or more identified features may be further processed by computing device 4000 of CPM 6000. However, as noted above, in some embodiments, the set of images and the associated one or more identified features may be transmitted to one or more computing devices remote from CPM 6000, such as a server that collects sets of images and inferences associated with multiple instruments and implements image selection logic 1008, training logic 1010, user interface logic 1012, or a combination or subset thereof (as well as model performance logic 1014, promotion logic 1018, user interface logic 1016, or a combination or subset thereof, as appropriate). Also, in some embodiments, generation of the set of images, the one or more identified features, or both may be performed on one or more computing devices remote from the CPM 6000. Thus, the inclusion of a computing device 4000 in the CPM 6000 illustrated in FIG. 20 represents one contemplated embodiment of such a scientific instrument.
[0103] Returning to FIG. 19 , the user local computing device 5020 may be a computing device (e.g., according to any of the embodiments of the computing device 4000 discussed herein) that is in the vicinity of a user of the scientific instrument 5010. In some embodiments, the user local computing device 5020 may also be in the vicinity of the scientific instrument 5010, but need not be. For example, a user local computing device 5020 in a user's home or office is remote from the scientific instrument 5010 but capable of communicating with it such that the user can control and / or access data from the scientific instrument 5010 using the user local computing device 5020. In some embodiments, the user local computing device 5020 may be a laptop, a smartphone, or a tablet device. In some embodiments, the user local computing device 5020 may be a portable computing device.
[0104] The service local computing device 5030 may be a computing device (e.g., according to any of the embodiments of the computing device 4000 discussed herein) that is proximate to an entity that provides services to the scientific instrument 5010. For example, the service local computing device 5030 may be proximate to the manufacturer of the scientific instrument 5010 or a third party service company. In some embodiments, the service local computing device 5030 may communicate with the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., via a direct communication path 5008 or via multiple "indirect" communication paths 5008 as previously discussed) to receive data regarding the operation of the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., results of self-diagnostic tests of the scientific instrument 5010, calibration coefficients used by the scientific instrument 5010, measurements of sensors associated with the scientific instrument 5010, etc.). In some embodiments, the service local computing device 5030 can communicate with the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., via a direct communication path 5008 or multiple "indirect" communication paths 5008, as discussed above) and transmit data to the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., to update programmed instructions, such as firmware, in the scientific instrument 5010, to initiate performance of a test or calibration sequence in the scientific instrument 5010, to update programmed instructions, such as software, in the user local computing device 5020 or the remote computing device 5040, etc.).A user of the scientific instrument 5010 may utilize the scientific instrument 5010 or the user local computing device 5020 to communicate with the service local computing device 5030 to report problems with the scientific instrument 5010 or the user local computing device 5020, to request a technician visit to improve the operation of the scientific instrument 5010, to order consumables or replacement parts associated with the scientific instrument 5010, or for other purposes.
[0105] The remote computing device 5040 may be a computing device (e.g., according to any of the embodiments of the computing device 4000 discussed herein) that is located remotely from the scientific instrument 5010 and / or from the user local computing device 5020. In some embodiments, the remote computing device 5040 may be included in a data center or other large-scale server environment. In some embodiments, the remote computing device 5040 may include network-attached storage (e.g., as part of the storage device 5004). The remote computing device 5040 may store data generated by the scientific instrument 5010, perform analysis of data generated by the scientific instrument 5010 (e.g., according to programmed instructions), facilitate communication between the user local computing device 5020 and the scientific instrument 5010, and / or facilitate communication between the service local computing device 5030 and the scientific instrument 5010. In some embodiments, the data triage logic 1002, the model promotion logic 1004, or a combination of a subset thereof, are implemented on the remote computing device 5040. For example, as described above, in some embodiments, the remote computing device 5040 receives data, such as a set of images and associated inferences generated via a machine learning model, from one or more scientific instruments 5010, and the remote computing device 5040 implements the image selection logic 1008, the training logic 1010, the user interface logic 1012, or a combination or subset thereof (as well as the model performance logic 1014, the promotion logic 1018, the user interface logic 1016, or a combination or subset thereof, as appropriate). Again, the functionality described herein as being performed via the support apparatus may be performed on a single device or distributed across multiple devices in various configurations.
[0106] In some embodiments, one or more of the elements of the scientific instrument support system 5000 illustrated in FIG. 19 may not be present. Furthermore, in some embodiments, multiples of various elements of the elements of the scientific instrument support system 5000 of FIG. 19 may be present. For example, the scientific instrument support system 5000 may include multiple user local computing devices 5020 (e.g., different user local computing devices 5020 associated with different users or at different locations). In another example, the scientific instrument support system 5000 may include multiple scientific instruments 5010, all in communication with a servicing local computing device 5030 and / or a remote computing device 5040. In such an embodiment, the servicing local computing device 5030 may monitor these multiple scientific instruments 5010, and the servicing local computing device 5030 may cause updates or other information to be "broadcast" to the multiple scientific instruments 5010 at the same time. Different scientific instruments 5010 in the scientific instrument support system 5000 may be located near each other (e.g., in the same room) or far from each other (e.g., different floors of a building, different buildings, different cities, etc.). In some embodiments, the scientific instruments 5010 may be connected to an Internet of Things (IoT) stack that allows command and control of the scientific instruments 5010 through web-based applications, virtual or augmented reality applications, mobile applications, and / or desktop applications. Any of these applications may be accessed by a user operating a user local computing device 5020 that is in communication with the scientific instruments 5010 through an intervening remote computing device 5040. In some embodiments, the scientific instruments 5010 may be sold by a manufacturer together with one or more associated user local computing devices 5020 as part of a local scientific instrument computing unit 5012.
[0107] In some embodiments, different ones of the scientific instruments 5010 included in the scientific instrument support system 5000 may be different types of scientific instruments 5010. In some such embodiments, the remote computing device 5040 and / or the user local computing device 5020 may combine data from different types of scientific instruments 5010 included in the scientific instrument support system 5000.
[0108] Thus, the embodiments described herein provide a continuous learning workflow for machine learning models. This workflow generally involves performing an automated data triage to automatically select useful images for training, testing, validation, and human review and annotation, and the datasets generated based on this automated data triage are used to train (i.e., retrain) the machine learning model. After this training (and associated testing), the machine learning model is used to generate future inferences that are used to control and operate the scientific instrument and related processes, such as, for example, sample preparation. Thus, as the machine learning model improves through this continuous learning workflow and adapts to changing processes, the resulting control and operation of the scientific instrument and related processes also improves. In some embodiments, this learning workflow uses data available at the customer's site, effectively transferring the learning workflow to the customer while minimizing human effort and required expertise in machine learning. In other words, the automated data triage optimizes human interaction in the learning workflow in an automated feedback loop.
[0109] As also described above, some embodiments provide automatic model promotion (e.g., as part of a workflow including automatic data triage, or separately from automatic data triage). Model promotion may be based on training loss or process-specific algorithms that consider one or more performance metrics of the machine learning models (e.g., generated based on one or more offline tests) and, if necessary, compare such performance metrics across available models to identify the best performing or optimal model. In some embodiments, multiple steps or stages of promotion may be used to sort the various models available, and model promotion integrates the machine learning models into laboratory processes (e.g., testing processes, production processes, etc.). By configuring promotion criteria, customers control the level of automatic model promotion to best suit their confidence and needs, without requiring expertise in machine learning. Thus, the automatic model promotion process identifies optimized models and deploys them to scientific instruments in a reliable and observable manner through customized levels of human intervention (e.g., tracking deployed models to define where, when, and which models are running).
[0110] Also, as noted above, although embodiments are described herein with respect to one or more particular scientific instruments (e.g., CPMs) and particular machine learning inferences (e.g., LIT runs), the methods and systems described herein are not limited in application to any particular scientific instrument or to any particular machine learning inferences. Rather, the methods and systems described herein can be used to provide training workflows and any model promotion workflows for machine learning models used by different types of scientific instruments and to generate different types of inferences.
[0111] According to one example embodiment disclosed above, for example in relation to any one or any combination of some or all of Figures 1-20, an apparatus is provided that includes feature identification logic that uses a machine learning model to generate one or more identified features in images of a set of images acquired via the scientific instrument; image selection logic that determines whether the set of images meets one or more selection criteria and, in response to a determination that the set of images meets the one or more selection criteria, assigns the set of images, including the one or more identified features, to a training dataset; and training logic that retrains the machine learning model using the training dataset.
[0112] In some embodiments of the above apparatus, the scientific instrument includes a charged particle microscope.
[0113] In some embodiments of any of the above apparatus, at least one of the image selection logic and the training logic is implemented by a computing device remote from the scientific instrument.
[0114] In some embodiments of any of the above apparatus, at least one of the image selection logic and the training logic is implemented in a scientific instrument.
[0115] In some embodiments of any of the above apparatus, the one or more identified features include a line indicating end feature.
[0116] In some embodiments of any of the above devices, the image selection logic determines whether the set of images meets one or more selection criteria by generating a metric related to the one or more identified features, and the image selection logic determines that the set of images meets the one or more selection criteria in response to the metric meeting a predetermined threshold.
[0117] In some embodiments of any of the above devices, the metric is based on at least one slope selected from the group consisting of a plot representing the number of features identified in each image in the set of images, a plot representing feature regions identified in each image in the set of images, and a plot representing the feature distance for each image in the set of images.
[0118] In some embodiments of any of the above devices, the one or more selection criteria may include predetermined criteria regarding characteristics of the one or more identified features, and the image selection logic determines whether the set of images satisfies the one or more selection criteria by comparing the predetermined criteria to identify anomalies in the one or more identified features.
[0119] In some embodiments of any of the above devices, the predetermined criteria regarding the characteristics of the one or more identified features includes at least one selected from the group consisting of a predetermined reference size of the one or more identified features, a predetermined reference number of the one or more identified features, a predetermined reference position of the one or more identified features, a predetermined reference shape of the one or more identified features, and a predetermined reference distance between two of the one or more identified features.
[0120] In some embodiments of any of the above devices, the image selection logic determines whether the set of images meets one or more selection criteria by comparing predetermined criteria to characteristics of one or more identified features within a single image of the set of images.
[0121] In some embodiments of any of the above devices, the image selection logic determines whether the set of images meets one or more selection criteria by comparing predetermined criteria to representative characteristics of one or more identified features in a plurality of images included in the set of images.
[0122] In some embodiments of any of the above devices, the representative characteristic comprises at least one selected from the group consisting of a mean value of the characteristic over the multiple images, an average of the characteristic over the multiple images, a median of the characteristic over the multiple images, a standard deviation of the characteristic over the multiple images, and a slope of a plot of the characteristic over the multiple images.
[0123] In some embodiments of any of the above devices, the predetermined criteria is user-defined.
[0124] In some embodiments of any of the above devices, the one or more selection criteria include characteristics of the one or more identified features, and the image selection logic determines whether a set of images satisfies the one or more selection criteria by identifying patterns of the characteristics across multiple sets of images.
[0125] In some embodiments of any of the above devices, the characteristics of the one or more identified features include at least one selected from the group consisting of a size of the one or more identified features, a number of the one or more identified features, a position of the one or more identified features, a shape of the one or more identified features, and a distance between two of the one or more identified features.
[0126] In some embodiments of any of the above apparatus, the pattern of the property includes a variation of the property across the sets of images.
[0127] In some embodiments of any of the above apparatus, the pattern of the characteristic comprises a change in the characteristic across the sets of images that exceeds a predetermined threshold.
[0128] In some embodiments of any of the above devices, the predetermined threshold is user-defined.
[0129] In some embodiments of any of the above apparatus, the one or more selection criteria include user-defined rules based on characteristics of the one or more identified features.
[0130] In some embodiments of any of the above devices, the one or more selection criteria includes random selection.
[0131] In some embodiments of any of the above apparatus, the random selection defines a predetermined frequency for including the set of images in the training data set.
[0132] In some embodiments of any of the above devices, the random selection is user-defined.
[0133] In some embodiments of any of the above devices, the one or more identified features include one or more first identified features of the first set of images, and the image selection logic excludes a second set of images from the training dataset that includes one or more second identified features of the second set of images.
[0134] In some embodiments of any of the above devices, the training dataset includes at least one selected from the group consisting of a retraining dataset, a testing dataset, a validation dataset, and an annotation dataset.
[0135] In some embodiments of any of the above apparatus, the training dataset includes an annotation dataset, and the image selection logic provides a user interface for receiving user annotations for images included in the annotation dataset.
[0136] In some embodiments of any of the above devices, the training dataset includes an annotation dataset, and the image selection logic provides a user interface and, in response to receiving an instruction through the user interface, assigns the set of images to at least one selected from the group consisting of a retraining dataset, a testing dataset, and a validation dataset.
[0137] In some embodiments of any of the above devices, the training dataset includes an annotation dataset, and the image selection logic provides a user interface and, in response to receiving an instruction through the user interface, excludes a set of images from at least one selected from the group consisting of a retraining dataset, a testing dataset, and a validation dataset.
[0138] In some embodiments of any of the above devices, the training dataset includes an annotation dataset, and the image selection logic, in response to assigning the set of images to the annotation dataset, generates and transmits a link selectable by a user to access the set of images assigned to the annotation dataset within a user interface.
[0139] In some embodiments of any of the above apparatus, the training logic retrains the machine learning model using the training dataset in response to the trigger event.
[0140] In some embodiments of any of the above devices, the trigger event includes at least one selected from the group consisting of a number of user-annotated images included in the training dataset, an increase in the size of the training dataset, an increase in the number of user-annotated images with respect to a given feature in the training dataset, availability of one or more training resources, and manual initiation.
[0141] According to another exemplary embodiment disclosed above, for example in relation to any one or any combination of some or all of Figures 1-20, there is provided a method executed via a computing device for providing scientific instrument support, the method including: receiving one or more selection criteria; receiving one or more identified features in a set of images acquired via the scientific instrument, the one or more identified features being generated using a machine learning model; determining whether the set of images satisfies the one or more selection criteria; and in response to a determination that the set of images satisfies the one or more selection criteria, including the set of images including the one or more identified features in a training dataset; and retraining the machine learning model using the training dataset.
[0142] In some embodiments of the above method, the one or more identified features in the set of images include one or more first identified features in a first set of images, and the method further includes receiving one or more second identified features in a second set of images acquired via the scientific instrument, where the one or more second identified features are generated using a machine learning model; providing the first set of images and the one or more first identified features to a user interface; providing the second set of images and the one or more second identified features to the user interface; in response to receiving a first instruction through the user interface, excluding the first set of images from the training dataset; and in response to receiving a second instruction through the user interface, including the second set of images in the training dataset.
[0143] In some embodiments of any of the above methods, the one or more selection criteria include one or more first selection criteria and the one or more identified features of the set of images include one or more first identified features of the first set of images, the method further including: receiving one or more second selection criteria; receiving one or more second identified features in a second set of images acquired via the scientific instrument, the one or more second identified features generated using a machine learning model; determining whether the second set of images satisfies the one or more second selection criteria; and in response to a determination that the second set of images satisfies the one or more second selection criteria, including the second set of images including the one or more second identified features in the training dataset.
[0144] In some embodiments of any of the above methods, the one or more identified features in the set of images include one or more first identified features in the first set of images, and the method further includes receiving one or more second identified features in a second set of images acquired via the scientific instrument, where the one or more second identified features are generated using a machine learning model; providing the second set of images and the one or more second identified features to a user interface; receiving annotations associated with the second set of images through the user interface; and including the second set of images including the annotations in a training dataset.
[0145] In accordance with yet another exemplary embodiment disclosed above, for example in relation to any one or any combination of some or all of Figures 1-20, there is provided one or more non-transitory computer readable media having instructions that, when executed by one or more processing devices of a scientific instrument support apparatus, cause the support apparatus to perform any of the methods described above.
[0146] According to another example embodiment disclosed above, for example in relation to any one or any combination of some or all of Figures 1-20, an apparatus is provided that includes feature identification logic that generates, for each of a plurality of machine learning models, one or more identified feature sets in a charged particle microscope image dataset using the machine learning models; model performance logic that generates, for each of the plurality of machine learning models, one or more performance measures; and model promotion logic that deploys a particular machine learning model to a plurality of scientific instruments based on the performance measures of the plurality of machine learning models.
[0147] According to another exemplary embodiment disclosed above, for example in relation to any one or any combination of some or all of Figures 1-20, an apparatus is provided, the apparatus including: feature identification logic for generating, for each of a plurality of machine learning models, one or more identified Feature sets in a charged particle microscopy image dataset using the machine learning models; first interface logic for generating a first interface having a first access permission for displaying the charged particle microscopy image dataset and the one or more identified Feature sets; model performance logic for generating, for each of the plurality of machine learning models, one or more performance measures; and a second interface for generating, for each of the plurality of machine learning models, a second interface having a second access permission, different from the first access permission, for displaying the one or more performance measures.
[0148] According to another exemplary embodiment disclosed above, for example in relation to any one or any combination of some or all of Figures 1-20, an apparatus is provided, the apparatus including model promotion logic receiving a first indication of one or more model promotion criteria; and model performance logic generating one or more performance measures for each of a plurality of machine learning models, each of the plurality of machine learning models for generating one or more identified feature sets in a charged particle microscope image dataset, and the model promotion logic deploying a particular machine learning model to a charged particle microscope for use in feature identification in a subsequently acquired charged particle microscope image dataset based on the performance measures of the plurality of machine learning models and the model promotion criteria.
[0149] Various features and advantages of the embodiments are set forth in the following claims.
Claims
1. A scientific instrument support device, comprising: feature identification logic for generating one or more identified features in images of the set of images acquired via the scientific instrument using a machine learning model; image selection logic that determines whether the set of images satisfies one or more selection criteria and, in response to a determination that the set of images satisfies the one or more selection criteria, assigns the set of images, including the one or more identified features, to a training dataset; and training logic for retraining the machine learning model using the training dataset.
2. 10. The scientific instrument support apparatus of claim 1, wherein at least one of the image selection logic and the training logic are implemented by a computing device remote from the scientific instrument.
3. 3. The scientific instrument support system of claim 1, wherein the one or more identified features include a line indication end feature.
4. 4. The scientific instrument support device of claim 3, wherein the image selection logic determines whether the set of images meets the one or more selection criteria by generating a metric for the one or more identified features, and the image selection logic determines that the set of images meets the one or more selection criteria in response to the metric meeting a predetermined threshold.
5. 5. The scientific instrument support apparatus of claim 4, wherein the metric is based on at least one slope selected from the group consisting of a plot representing a number of features identified in each image in the set of images, a plot representing feature regions identified in each image in the set of images, and a plot representing feature distances for each image in the set of images.
6. 3. The scientific instrument support apparatus of claim 1, wherein the one or more selection criteria include predetermined criteria regarding characteristics of the one or more identified features, and the image selection logic determines whether the set of images meets the one or more selection criteria by comparing the one or more identified features to the predetermined criteria and identifying anomalies in the one or more identified features.
7. 7. The scientific instrument support apparatus of claim 6, wherein the predetermined criteria regarding the characteristics of the one or more identified features includes at least one selected from the group consisting of a predetermined reference size of the one or more identified features, a predetermined reference number of the one or more identified features, a predetermined reference position of the one or more identified features, a predetermined reference shape of the one or more identified features, and a predetermined reference distance between two of the one or more identified features.
8. 3. The scientific instrument support device of claim 1, wherein the one or more selection criteria include characteristics of the one or more identified features, and the image selection logic determines whether the set of images meets the one or more selection criteria by identifying patterns of the characteristics across multiple sets of images.
9. 9. The scientific instrument support apparatus of claim 8, wherein the characteristics of the one or more identified features include at least one selected from the group consisting of a size of the one or more identified features, a number of the one or more identified features, a position of the one or more identified features, a shape of the one or more identified features, and a distance between two of the one or more identified features.
10. 3. The scientific instrument support system of claim 1, wherein the one or more identified features include one or more first identified features of a first set of images, and the image selection logic excludes the second set of images from the training data set that includes one or more second identified features of a second set of images.
11. 3. The scientific instrument support device of claim 1, wherein the training data set includes an annotation data set, and the image selection logic provides a user interface and, in response to receiving an instruction through the user interface, assigns the set of images to at least one selected from the group consisting of a retraining data set, a testing data set, and a validation data set.
12. 3. The scientific instrument support device of claim 1, wherein the training dataset includes an annotation dataset, and the image selection logic provides a user interface and, in response to receiving an instruction through the user interface, excludes the set of images from at least one selected from the group consisting of a retraining dataset, a testing dataset, and a validation dataset.
13. 3. The scientific instrument support device of claim 1 or 2, wherein the training dataset includes an annotation dataset, and the image selection logic, in response to assigning the set of images to the annotation dataset, generates and transmits a link selectable by a user to access the set of images assigned to the annotation dataset in a user interface.
14. The scientific instrument support apparatus of claim 1 or 2, wherein the training logic is responsive to a trigger event to retrain the machine learning model using the training data set.
15. 15. The scientific instrument support apparatus of claim 14, wherein the trigger event comprises at least one selected from the group consisting of a number of user-annotated images included in the training dataset, an increase in the size of the training dataset, an increase in the number of user-annotated images for a given feature in the training dataset, availability of one or more training resources, and manual initiation.
16. 1. A method for providing scientific instrument support, performed via a computing device, the method comprising: Receiving one or more selection criteria; receiving one or more identified features in a set of images acquired via a scientific instrument, the one or more identified features being generated using a machine learning model; determining whether said set of images satisfies said one or more selection criteria; in response to determining that the set of images satisfies the one or more selection criteria, including the set of images, including the one or more identified features, in a training dataset; and retraining the machine learning model using the training dataset.
17. the one or more identified features in the set of images include one or more first identified features in a first set of images; receiving one or more second identified features in a second set of images acquired via the scientific instrument, the one or more second identified features being generated using the machine learning model; and providing the first set of images and the one or more first identified features to a user interface; providing the second set of images and the one or more second identified features to the user interface; and excluding the first set of images from the training data set in response to receiving a first instruction through the user interface; 17. The method of claim 16, further comprising: in response to receiving a second instruction through the user interface, including the second set of images in the training data set.
18. the one or more selection criteria include the one or more first selection criteria, and the one or more identified features of the set of images include one or more first identified features of a first set of images; receiving one or more second selection criteria; receiving one or more second identified features in a second set of images acquired via the scientific instrument, the one or more second identified features being generated using the machine learning model; and determining whether the second set of images meets the one or more second selection criteria; 17. The method of claim 16, further comprising: in response to determining that the second set of images meets the one or more second selection criteria, including the second set of images, comprising the one or more second identified features, in the training data set.
19. the one or more identified features in the set of images include one or more first identified features in a first set of images; receiving one or more second identified features in a second set of images acquired via the scientific instrument, the one or more second identified features being generated using the machine learning model; and providing the second set of images and the one or more second identified features to a user interface; receiving, through the user interface, annotations associated with the second set of images; The method of claim 16 , further comprising: including the second set of images, including the annotations, in the training dataset.
20. 20. One or more non-transitory computer-readable media having instructions that, when executed by one or more processing devices of a support apparatus for the scientific instrument, cause the support apparatus to perform the method of claim 16.
Citation Information
Patent Citations
Image evaluation apparatus and method
JP2020119135A
