Classifying microscopic components of physical sample

Abstract

Systems for classifying microscopic components of a physical sample, and related methods, computing devices, and computer-readable media, are disclosed herein. For example, in some embodiments, a method for classifying microscopic components of a physical sample may include generating a set of regions of interest (ROI) in an image representing the physical sample, where the image is generated by a microscope system using a first analysis pattern; generating an initial classification for the ROI by applying a trained machine learning model to at least a portion of the image associated with the ROI; generating a confidence score associated with the initial classification; and when the confidence score of the initial classification of the ROI does not meet a set of confidence criteria, causing the microscope system to reanalyze at least the portion of the sample associated with the ROI using a second analysis pattern different from the first analysis pattern.

Description

BACKGROUND

[0001] There are many types of analytical instruments that can generate data about microscopic features of a sample. Different types of such instruments can use different physical principles to generate data about a sample. BRIEF DESCRIPTION OF DRAWINGS

[0002] Embodiments will be readily understood by the following detailed description in conjunction with the accompanying drawings. Like reference numerals designate like structural elements throughout the drawings. Embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings.

[0003] Figure 1 is a block diagram of an example sample analysis module for performing sample analysis operations in accordance with various embodiments.

[0004] Figure 2 An example of an image in which a plurality of regions of interest (ROIs) have been identified is illustrated in accordance with various embodiments.

[0005] Figure 3 An example result of energy dispersive spectroscopy (EDS) analysis of a portion of a sample associated with a particular ROI is illustrated in accordance with various embodiments.

[0006] Figure 4 is a flowchart of an example method of performing sample analysis operations in accordance with various embodiments.

[0007] Figure 5 is a flowchart of an example method of generating a machine learning model for classifying microscopic components of a physical sample in accordance with various embodiments.

[0008] Figure 6 is a flowchart of a method of creating a new classification for a previously unclassified ROI in accordance with various embodiments.

[0009] Figure 7 is an example of a graphical user interface (GUI) that can be used to perform some or all of the sample analysis methods disclosed herein in accordance with various embodiments.

[0010] Figure 8 is a block diagram of an example computing device that can perform some or all of the sample analysis methods disclosed herein in accordance with various embodiments.

[0011] Figure 9 is a block diagram of an example sample analysis system in which some or all of the sample analysis methods disclosed herein can be performed in accordance with various embodiments. DETAILED DESCRIPTION

[0012] Disclosed herein are systems for classifying microscopic components of physical samples, as well as related methods, computing devices, and computer-readable media. For example, in some embodiments, a method for classifying microscopic components of a physical sample can include generating a set of regions of interest (ROIs) in an image representing the physical sample, where the image is generated by a microscopy system using a first analysis mode; generating an initial classification for a ROI by applying a trained machine learning model (ML) to at least a portion of the image associated with the ROI; generating a confidence score associated with the initial classification of the ROI; and causing the microscopy system to reanalyze at least the portion of the sample associated with the ROI using a second analysis mode different from the first analysis mode when the confidence score of the initial classification of the ROI does not satisfy a set of confidence criteria.

[0013] As discussed in further detail below, many scientific and industrial applications can benefit from accurate classification of microscopic components of samples (e.g., to avoid using“dirty” components in an assembly process, to ensure that materials have sufficient quality before further processing, to facilitate effective law enforcement through rapid analysis of crime scene samples, etc.). In many such applications, generating accurate classifications as quickly as possible is critical to enable timely use of the information. The need for speed is particularly important in a large number of analysis applications in which there are many samples that need to be analyzed (e.g., in some automotive manufacturing processes, every tenth sample needs to be analyzed), and longer processing times can result in undesirable backlogs. In some applications, the analysis must be non-destructive to the sample, must be highly repeatable (e.g., 93-99% of particles are detected and matched), and / or must be repeatable when using different microscopy systems (e.g., such that classification results using one microscope match classification results using a different microscope, with some applications specifying + / - 8% or less error tolerance as part of a field acceptance test).

[0014] Disclosed herein are sample analysis techniques and systems that enable accurate classification of microscopic components of samples much more quickly than conventional classification systems. The techniques and systems disclosed herein can leverage machine learning techniques and can be implemented without requiring human users to set aside additional time to laboriously create a set of training data as ML tools typically require. Moreover, various embodiments of the techniques and systems disclosed herein provide autonomous or semi-autonomous processes for developing ML-based classifiers, enabling model training and deployment with little or no human effort. Once in place, the ML-based techniques and systems disclosed herein can enable classification of components of samples much more quickly than conventional classification systems, with the potential for orders-of-magnitude improvements in evaluation time of samples.

[0015] Accordingly, the sample analysis implementations disclosed herein can achieve improved performance relative to conventional approaches. In particular, the implementations disclosed herein provide improvements to scientific instrumentation technology (e.g., improvements to computer technology supporting such scientific instrumentation, among other improvements). Various of the implementations disclosed herein can improve upon conventional approaches to achieve technical advantages in faster classification of microscopic components of physical samples through autonomous or semi-autonomous training and deployment of ML models to at least partially replace time-consuming conventional analysis techniques and achieve higher throughput. Such technical advantages cannot be achieved by conventional approaches and traditional methods, and all users of systems comprising such implementations can benefit from these advantages. Accordingly, the technical features of the implementations disclosed herein are decidedly unconventional in the field of sample analysis, as are the combinations of features of the implementations disclosed herein. As discussed further herein, various aspects of the implementations disclosed herein can improve the functionality of the computer itself; e.g., a computing system analyzing sample analysis data. The computing and user interface features disclosed herein are directed not only to the collection and comparison of information, but also to the application of new analysis and expertise to change the operation of systems that employ microscopic feature classification as part of a scientific or industrial process. Accordingly, the present disclosure introduces functionality that neither conventional computing equipment nor humans can perform.

[0016] Accordingly, the implementations of the present disclosure can serve any of a number of technical purposes, such as controlling a particular technical system or method; determining how to control a machine or process based on measurements (e.g., classification of microscopic components of a sample); digital image enhancement or analysis; and providing faster processing of analysis instrument data.

[0017] Accordingly, the implementations disclosed herein provide improvements to analysis instruments and sample analysis techniques (e.g., improvements to computer technology supporting sample analysis, among other improvements).

[0018] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration embodiments in which the subject matter disclosed herein can be practiced. It is to be understood that other embodiments can be utilized and structural or logical changes can be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense as the scope of the disclosure is defined by the appended claims.

[0019] Various operations can be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the claimed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations can not be performed in the order of presentation. Operations described can be performed in a different order than the described embodiment. Various additional operations can be performed and / or described operations can be omitted in additional embodiments.

[0020] For purposes of this disclosure, the phrase "A and / or B" and "A or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" and "A, B, or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). Although some elements can be shown in singular form (e.g., "processing device"), any appropriate number of instances of such elements can be represented by a single instance of that element, and vice versa. For example, a set of operations described as being performed by a processing device can be implemented by different processing devices performing different ones of the operations. As used in this document, the phrase "based on" is understood to mean "based, at least in part, on," unless otherwise indicated.

[0021] This specification uses the phrases "embodiment," "various embodiments," and "some embodiments," each of which can refer to one or more of the same or different embodiments. Furthermore, the terms "comprising," "containing," "having," and the like, as used with respect to an embodiment of the present disclosure, are synonymous with "including." The phrase "between X and Y" when used in a context to describe a range means a range that includes X and Y. As used herein, "device" can refer to any individual device, a collection of devices, a part of a device, or a collection of parts of devices. The drawings are not necessarily to scale.

[0022] Figure 1 is a block diagram of a sample analysis module 1000 for performing sample analysis operations in accordance with various embodiments. The sample analysis module 1000 can be implemented by circuitry, such as a programmed computing device (e.g., including electrical and / or optical components). The logical components of the sample analysis module 1000 can be included in a single computing device or can be distributed across multiple computing devices in communication with one another as appropriate. Examples of computing devices that can implement the sample analysis module 1000, alone or in combination, are discussed herein with respect to the computing device 4000 of Figure 8 Examples of systems of interconnected computing devices in which the sample analysis module 1000 can be implemented across one or more of the computing devices are discussed herein with respect to the sample analysis system 5000 of Figure 9 Examples of systems of interconnected computing devices in which the sample analysis module 1000 can be implemented across one or more of the computing devices are discussed herein with respect to the sample analysis system 5000 of

[0023] The sample analysis module 1000 can include an instrument interface logic 1002, a region of interest (ROI) logic 1004, a classification logic 1006, an evaluation logic 1008, and a user interface logic 1010. As used herein, the term "logic" can include a device that performs a set of operations associated with that logic. For example, any of the logic elements included in the sample analysis module 1000 can be implemented by one or more computing devices programmed with instructions to cause one or more processing devices of that computing device to perform the associated set of operations. In particular embodiments, a logic element can include one or more non-transitory computer-readable media having instructions thereon that, when executed by one or more processing devices of one or more computing devices, cause the one or more computing devices to perform the associated set of operations. As used herein, the term "module" can refer to a collection of one or more logic elements that together perform the functions associated with the module. Different ones of the logic elements in a module can take the same form or can take different forms. For example, some of the logic elements in a module can be implemented by a programmed general purpose processing device, while other logic elements in the module can be implemented by an application specific integrated circuit (ASIC). In another example, different ones of the logic elements in a module can be associated with different sets of instructions executed by one or more processing devices. A module can not include all of the logic elements depicted in the associated figure; for example, when that module is to perform a subset of the operations discussed herein with reference to that module, that module can include a subset of the logic elements depicted in the associated figure.

[0024] The instrument interface logic component 1002 can allow the sample analysis module 1000 to communicate with one or more scientific instruments, such as one or more microscopes (e.g., charged particle microscopes). In some embodiments, this communication can include receiving data from one or more scientific instruments (e.g., data collected using one or more analysis modes of the scientific instrument) and / or providing commands or instructions to one or more scientific instruments (e.g., to cause a microscope to image a sample using a specified analysis mode and on a specified ROI). In various embodiments disclosed herein, the scientific instruments that generate data to be analyzed by the sample analysis module 1000 can have a variety of possible acquisition modes. These modes can generate different kinds of data, and the selection of one mode or another for analysis can require a balancing of competing factors. For example, some acquisition modes can generate images with higher information density than other modes, but at the cost of longer acquisition times, greater power requirements, greater risk of damaging the sample, or other factors. For example, some microscope systems can be configured to be capable of performing in a backscattered electron detection (BSED) analysis mode, and also capable of performing in an energy dispersive spectroscopy (EDS) analysis mode; EDS can provide additional information about the elemental composition of a sample than BSED, but can take more time (e.g., more than 1000 times longer to image the same region).

[0025] BSED and EDS can be used herein as an example of a pair of analysis types that provide different kinds or amounts of information about a sample, and where one analysis type is more resource-intensive than the other analysis type, but there are many other pairs of analysis types for which the innovative systems and methods disclosed herein can be applied. For example, a first, lower resource-intensive analysis type can include any of secondary electron detector (SED) analysis, low vacuum detector (LVD) analysis, circular backscatter detector (CBS) analysis, Everhart-Thornley detector (ETD) analysis, BSED analysis, specific detector analysis (e.g., from in-lens detector Tl or T2 and / or in-column detector T3), visible light analysis, infrared light analysis, or ultraviolet light analysis, and a second, higher resource-intensive analysis type can include any of EDS analysis, electron backscatter diffraction (EBSD) analysis, electron energy loss spectroscopy (EELS) analysis, cathodoluminescence (CL) analysis, or wavelength dispersive X-ray spectroscopy (WDS) analysis.

[0026] The ROI logic 1004 can identify one or more ROIs in an image of a physical sample. The image can be constructed from data generated by a scientific instrument (e.g., a charged particle microscope), and can represent use of the scientific instrument to one or more modes of analysis. As used herein, an “image” includes two-dimensional data representations and suitable higher-dimensional data representations. For example, a multi-channel image can include multiple two-dimensional images, one for each channel (e.g., three two-dimensional images for a red-green-blue (RGB) image capture device). In another example, when an image is generated from data from multiple detectors (e.g., a BSED and a SED), the resulting image can be a three-dimensional matrix (represented by, e.g., (w, h, x) coordinates, where w is width, h is height, and x is a data vector corresponding to a different detector). An ROI can identify a particular portion of an image corresponding to a feature of interest of a physical sample, where the particular feature of interest of a particular sample depends on the nature of the sample and the purpose of the sample analysis. In some embodiments, a single ROI can correspond to a single particle in a physical sample. In some embodiments, a single ROI can correspond to a cluster of particles in a physical sample (e.g., a beam of a steel sample). In some embodiments, a single ROI can correspond to a structure of interest in a biological sample (e.g., a nucleus of a cell).

[0027] The ROI logic 1004 can identify ROIs in an image using any suitable technique. In some embodiments, the ROI logic 1004 can apply conventional segmentation techniques, such as machine learning (ML)-based segmentation techniques or any other computer vision-based techniques known in the art (e.g., thresholding, edge detection, etc.) to identify ROIs. In some embodiments in which the ROI logic 1004 is to determine ROIs in a BSED image, the ROI logic 1004 can employ a BSED thresholding technique in which pixels of a certain intensity are used to identify ROIs, as known in the art.

[0028] Figure 2 An example of an image 1100 in which a plurality of ROIs 1102 have been identified is illustrated, as shown by the highlighted regions. The image 1100 can represent only a portion of a sample, and can be one of many images 1100 tiled or otherwise arranged to represent a field of view. In some embodiments, the image 1100 can be a BSED image. Figure 2 Only a few of the ROIs 1102 identified in the image 1100 are labeled. The image 1100 (which can be, e.g., a BSED image) can be generated or received by the instrument interface logic 1002, and the ROIs 1102 are identified by the ROI logic 1004. Figure 3An example result 1104 of an EDS analysis of a portion of a sample associated with a particular ROI 1102 identified in the image 1100 is illustrated. The result 1104 can include peaks 1106 corresponding to different elements present in the portion of the sample.

[0029] The classification logic component 1006 can store one or more ML models that can be used to generate a classification of a particular ROI identified by the ROI logic component 1004. The classification logic component 1006 can apply the ML models to the ROI to generate such a classification. The classification logic component 1006 can generate new ML models based on previously performed classifications (e.g., manually or using a rules-based process). Reference is made below to Figure 5 An example method for creating new ML models for classifying ROIs in images of physical samples is discussed. In some embodiments, the classification logic component 1006 can retrain or otherwise update stored ML models (e.g., based on additional available data). In some embodiments, the classification logic component 1006 can deploy one or more ML models to computing devices connected to or included in scientific instruments, such that those ML models can be run on those computing devices on data generated by the associated scientific instruments. In some embodiments, the ML models applied by the classification logic component 1006 can be received by the classification logic component 1006 from a central server (e.g., the remote computing device 5040 discussed below) that can be configured to deploy ML models to multiple microscope systems. In other embodiments, the classification logic component 1006 itself can deploy one or more ML models to other microscope systems. Figure 9

[0030] The set of possible classifications (e.g., two or more) that can be generated by the ML models of the classification logic component 1006 can depend on the particular application (e.g., law enforcement, automotive, battery manufacturing, steel processing, etc.) and how the ML models are trained. For example, in some law enforcement applications that analyze materials, the set of possible classifications can include gunshot residue (GSR) instead of GSR. In some automotive applications that can evaluate cleanliness of parts prior to assembly, the set of possible classifications can include worn (which can include characterization of particles of silicon carbide and other materials) and soft (which can include characterization of aluminum and other materials).

[0031] ​The architecture of the ML model used by the classification logic component 1006 can take any of a variety of forms. In some embodiments, the ML model can take the form of a classifier model as known in the art, such as a fully connected neural network. The input to the ML model can be a two-dimensional image or a one-dimensional vector. In some embodiments, the two-dimensional image provided as input to the ML model can be a BSED or other image output by the first analysis mode of the ROI. In some embodiments, the one-dimensional vector provided as input to the ML model can include a set of morphological parameters of the ROI generated based on the results of the first analysis mode. For example, a binary mask can be applied to the BSED or other image to isolate the ROI, and then morphological attributes of the ROI can be computed using techniques known in the art; then, for a particular ROI, the input to the ML model can be a vector of its morphological properties. Examples of morphological properties that can be computed can include, but are not limited to, size, area, circularity, sharpness, elongation, aspect ratio, shape factor, perimeter, area, brightness level, void count, void area, or skeleton length. In some embodiments, the input to the ML model can include both two-dimensional image data and computed morphological properties of the ROI. The output of the ML model can be a classification of the ROI (e.g., GSR or non-GSR, worn or soft, etc.) based on the number and variety of classifications used in a particular application.

[0032] The classification logic component 1006 can also include user-defined rules (sometimes included in a “rules file”) to be applied to data generated by the various analysis modes to arrive at a classification of the associated ROI, distinct from the ML model that generates such a classification. These additional rules can be used when further information is available about a ROI (e.g., when multiple different analysis modes are used to image a portion of the physical sample corresponding to the ROI). For example, if a physical sample is imaged using EDS, and the EDS provides information about the elemental composition of the portion of the physical sample corresponding to the ROI, the classification logic component can store rules in which the elemental composition information (potentially in conjunction with properties of the ROI, such as its size, roughness, etc.) can be compared to determine an appropriate classification of the ROI. Other analysis modes can provide other kinds of information that can be part of the set of classification rules (e.g., crystal structure rotation information provided by EBSD).

[0033] In some embodiments, the ML model implemented by the classification logic component 1006 can be trained using prior classifications (e.g., input-output pairs including images / morphology data for individual ROIs and classifications of those ROIs). This training can begin when the module 1000 is first deployed to a particular site for a particular application, or when the application area is known and training data is available. In some embodiments, the ML model can be refined or initially trained based on data that becomes available to the module 1000 after deployment. For example, a microscopy system can initially generate BSED and EDS data for each ROI (when both analysis modes are available), generate classifications for those ROIs using user-defined rules, and then use this data (with BSED and BSED-derived data as input and classifications as output in a training set) to initially train the ML model to perform the classification or improve the performance of a previously trained ML model. In this way, as the classification performance of the ML model improves (as determined by the evaluation logic component 1008, as discussed below), fewer ROIs can need to be analyzed with EDS for accurate classifications to be determined, resulting in significant time savings.

[0034] In some embodiments, a classification of a ROI can be “unclassified,” meaning that the proper classification of the ROI is unknown. This can occur when the ML model of the classification logic component 1006 outputs an “unclassified” result (e.g., when confidence in a particular classification is low, as determined by the evaluation logic component 1008, as discussed below), when the ROI cannot be reanalyzed using a second analysis mode (e.g., when a particular microscopy system is not configured to analyze using the second mode), and / or when the analysis results following the second analysis mode (e.g., according to stored rules applied by the classification logic component 1006) do not correspond to any known classification. In some such instances, when a classification of a ROI is unclassified, the classification logic component 1006 can store an “unclassified” classification for the ROI, and can update the classification of the ROI upon notification from a central server that a classification is available. The classification can become available in any of a variety of circumstances, such as improving the performance of the ML model to account for previously unclassified ROIs, creating a new ML model to properly classify previously unclassified ROIs, and / or creating new rules that can dispose of previously unclassified ROIs. Examples of methods for creating new classifications for previously unclassified ROIs are discussed below with reference to Figure 6

[0035] ​The evaluation logic 1008 can evaluate the classification of the ROI generated by the classification logic 1006 to determine whether the confidence in the classification is high enough to accept the classification for the ROI. In some embodiments, the classification logic 1006 can generate a confidence score along with its classification using techniques known in the art of machine learning, and the evaluation logic 1008 can compare the confidence score to a predetermined threshold to determine whether the classification of the ROI should be accepted (e.g., stored, included in a report assembled by the user interface logic 1010, etc.) or whether the ROI should be re-analyzed (e.g., using a different, more resource-intensive analysis modality like EDS) to generate data that can be used to generate a higher confidence classification of the ROI. In some embodiments, the confidence score can be a value between 0 and 1, as is known in the art, although the range of the confidence score can be scaled to any desired range. The evaluation logic 1008 can compare the confidence score to a threshold (e.g., 0.95 or another value), and can accept classifications for which the confidence meets or exceeds the threshold (and can reject classifications in other ways).

[0036] In some embodiments, the evaluation logic 1008 can track the confidence of the ML model of the classification logic 1006 over time, and can monitor confidence trends that indicate that the performance of the ML model is decreasing or has fallen below a threshold for certain output classifications or for all classifications. If such a performance decrease condition is identified by the evaluation logic 1008, the evaluation logic 1008 can cause the classification logic 1006 to retrain the ML model on additional data in order to improve performance. Causing the classification logic 1006 to retrain can mean causing the instrument interface logic 1002 to collect more data using both the first analysis mode and the second analysis mode, user-defined rules can be applied to the data to generate classifications, and the resulting data and classifications can be provided to the classification logic 1006 to retrain the ML model.

[0037] User interface logic component 1010 can provide information to and / or receive information from human users of analysis module 1000. In some embodiments, user interface logic component 1010 can aggregate information about a physical sample into a report, which includes information about the ROI identified in an image of the physical sample. This report can be provided to the user for visual display, electronic transmission, or used by quality control or other systems to aid in making automated or semi-automated decisions regarding downstream processing or disposal of the physical sample or upstream parameters (e.g., whether previous processing steps were performed correctly, whether raw materials have appropriate properties, whether the quality of the manufactured material meets specified standards, etc.). In some embodiments where the systems and methods disclosed herein are used as part of a particle analysis (PA) workflow, the report output by user interface logic component 1010 can identify all identifiable particles in the physical sample and their associated classifications. Several examples of reports that can be generated by user interface logic component 1010 are discussed herein.

[0038] The sample analysis module 1000 can execute any of a variety of sample analysis methods. Figure 4 This is a flowchart of a method 2000 for performing sample analysis operations according to various implementation schemes. Specifically, method 2000 is a method for classifying the microscopic components of a physical sample. Although the operation of method 2000 (and other methods disclosed herein) may be illustrated with reference to specific implementation schemes disclosed herein (e.g., references herein...), Figure 1 The sample analysis module 1000 discussed in this article, and the references in this article Figure 7 The GUI 3000 discussed in this article is referenced. Figure 8 The computing device discussed is 4000 and / or referenced in this document. Figure 9 The sample analysis system discussed is 5000, but method 2000 (and other methods disclosed herein) can be used in any suitable setup to perform any suitable sample analysis operation. Figure 4 Operations are each instantiated once in a specific order, but can be reordered and / or repeated as needed and as appropriate (e.g., different operations can be executed in parallel where appropriate).

[0039] At 2002, a set of ROIs in an image representing a physical sample can be generated. The image itself can be generated by a microscope system using a first analysis mode. The ROI logic 1004 of the sample analysis module 1000 can perform the operation of 2002 based on data provided by the instrument interface logic 1002 of the sample analysis module 1000. In some embodiments, generating the set of ROIs at 2002 includes applying an ML-based segmentation technique to the image. The particular ROIs identified will depend on the application. For example, in some applications, a single ROI identified at 2002 can correspond to a single particle in the physical sample.

[0040] At 2004, an initial classification of the single ROI (of the set of ROIs generated at 2002) can be generated by applying a trained ML model to at least a portion of the image associated with the ROI. The classification logic 1006 of the sample analysis module 1000 can perform the operation of 2004. As noted above, the set of possible classifications (e.g., two or more) that can be generated at 2004 will depend on the particular application and how the ML model is trained. For example, in some law enforcement applications, the initial classification of the ROI at 2004 can be selected from a set of at least two classifications, and the set of at least two classifications includes GSR but not GSR. In some automotive applications, the initial classification of the ROI at 2004 can be selected from a set of at least two classifications, and the set of at least two classifications includes worn and soft. In some embodiments, the ML model applied at 2004 can be received by the classification logic 1006 from a central server (e.g., the remote computing device 5040 discussed below) that can be configured to deploy ML models to multiple microscope systems. In other embodiments, the classification logic 1006 itself can deploy the ML model to other microscope systems. Figure 9

[0041] At 2006, a confidence score associated with the initial classification (generated at 2004) can be generated. The classification logic 1006 of the sample analysis module 1000 can perform the operation of 2006, and in some particular embodiments, this operation can be performed in conjunction with or concurrently with the generation of the initial classification at 2004.

[0042] At 2008, a determination can be made as to whether the initial classification of the single ROI satisfies a set of one or more confidence criteria. For example, the evaluation logic 1008 of the sample analysis module 1000 can perform the operation of 2008. In some embodiments, the operation of 2008 can include comparing the confidence score (generated at 2006) to a threshold value; when the confidence score exceeds the threshold value, the confidence criterion can be satisfied.

[0043] ​If it is determined at 2008 that the initial classification satisfies the confidence criteria, the method 2000 can proceed to 2010 and the initial classification can be set as the final classification for the associated ROI. The user interface logic component 1010 of the sample analysis module 1000 can perform the operations of 2010.

[0044] If it is determined at 2008 that the initial classification does not satisfy the confidence criteria, the method 2000 can proceed to 2012 and the microscope system can be caused to reanalyze at least a portion of the sample associated with the ROI using a second analysis mode that is different from the first analysis mode (e.g., perform an EDS analysis after an initial BSED analysis). The instrument interface logic component 1002 of the sample analysis module 1000 can perform the operations of 2012. In some embodiments, the microscope system takes less time to image the portion of the sample associated with the ROI using the first analysis mode (the mode used to generate the image analyzed at 2002) than using the second analysis mode (the mode triggered at 2012 as a result of failing to satisfy the confidence criteria of 2008). For example, in some embodiments, a confidence score is generated associated with the initial classification, the first analysis mode can be or include BSED, and the second analysis mode can be or include EDS. Because another round of imaging using the second analysis mode can not be triggered unless the initial classification fails to satisfy the confidence criteria, it is expected that as long as the ML model used in the initial classification at 2004 has sufficient performance (e.g., once the ML model has been sufficiently trained), it is not necessary to reanalyze all of the ROIs generated at 2002, and thus the execution of the method 2000 can take less time to analyze the physical sample than a conventional method in which all of the ROIs are imaged using the second analysis mode.

[0045] In some embodiments, the method 2000 can further include, after reanalyzing the sample at 2012, generating a final classification for the ROI using data generated by the reanalysis. The classification logic component 1006 can include rules applied to the data generated by the second analysis mode (potentially in combination with the first image and / or attributes of the ROI) to arrive at a classification for the associated ROI. For example, if the second analysis mode is EDS, and reanalyzing the ROI with EDS provides information about the elemental composition of the portion of the physical sample corresponding to the ROI, this elemental composition information (potentially in combination with attributes of the ROI, such as its size, roughness, etc.) can be compared to a pre-stored set of rules to determine an appropriate classification for the ROI.

[0046] Once the method 2000 has been performed for all ROIs of a particular physical sample, the user interface logic 1010 can then report, store, electronically transmit, or otherwise use the final ROI classification for each of the ROIs. In some embodiments, the user interface logic 1010 can output a classification report that includes the final classification of individual ROIs of the set of ROIs. The classification report can include any other suitable information, such as the location of the individual ROIs, the morphological characteristics (e.g., area, circularity, or roughness) of the sample portion corresponding to the individual ROIs, the composition information (e.g., gunpowder, aluminum oxide, etc.) of the sample portion corresponding to the individual ROIs, and / or any other suitable information.

[0047] As described above, in some embodiments of the method 2000, the initial and / or final classification of a ROI can be “unclassified,” which means that the proper classification of the ROI is unknown (e.g., the classification was determined at 2008 not to satisfy the confidence criteria for the “accepted” classification). In some embodiments, the method 2000 can include the classification logic 100 storing the “unclassified” classification of the ROI when the classification of the ROI is unclassified, and can update the classification of the ROI upon notification that a classification from a central server is available.

[0048] As described above, in some embodiments, the sample analysis module 1000 can generate a new ML model. Figure 5 is a flowchart of a method 2100 of generating a ML model for classifying microscopic components of a physical sample, in accordance with various embodiments. In Figure 5 The operations in are each illustrated in a particular order, but the operations can be reordered and / or repeated as necessary and as appropriate (e.g., different operations can be performed in parallel, where appropriate).

[0049] At 2102, first data representing a sample can be received. The first data can be generated using a first analysis mode (e.g., of a microscope system). The instrument interface logic 1002 can perform the operation of 2102.

[0050] At 2104, a classification can be received. The classification can correspond to a ROI in the first data, the classification can have been determined using the second data and a second analysis mode (e.g., of the microscope system) different from the first analysis mode. Classification logic 1006 can perform the operations of 2104. In some embodiments, the classification can be generated manually by a human user or using a rules-based process. In some embodiments, the microscope system takes less time to image a portion of a sample using the first analysis mode than using the second analysis mode (e.g., the first analysis mode can be or include BSED and the second analysis mode can be or include EDS or another method that can generate elemental composition data for the portion of the sample). In some embodiments, a single ROI in the first data corresponds to a single particle in the physical sample. In some such embodiments, the classification received at 2104 can be selected from a set of at least two classifications (e.g., gunshot residue (GSR) versus GSR, worn and soft, etc.).

[0051] At 2106, an ML model can be trained using the first data and the classification to classify ROIs in the first analysis mode data. Classification logic 1006 can perform the operations of 2106. In some embodiments, training the ML model using the first data and the classification can include training the machine learning model using the first analysis data, morphological parameters (e.g., area, circularity, roughness, etc.) generated at least in part from the first analysis data, or a combination of both. In some embodiments, the ML model of 2106 can be configured to receive as input a two-dimensional image (e.g., one or more images included in the first data), a one-dimensional vector (e.g., a vector of morphological parameters), or a combination of both (e.g., represented as an array, concatenated with or otherwise combined with data in the one-dimensional vector of morphological parameters).

[0052] In some embodiments, the operations of 2106 can include generating a training performance score (e.g., by classification logic 1006) after training the ML model. In such embodiments, the training performance score can be compared to a predetermined training performance criterion (e.g., by evaluation logic 1008). If the training performance score does not satisfy the training performance criterion (e.g., the classification performance of the ML model is not sufficient), the ML model can be retrained based on additional first data and additional corresponding classification data (e.g., by classification logic 1006). If the training performance score satisfies the training performance criterion, the ML model can be provided (e.g., by classification logic 1006) to generate classifications for ROIs in additional images of the physical sample, where the additional images are generated by the microscope system using the first analysis mode. In this way, the ML model can not be deployed until its classification performance is sufficient.

[0053] In some embodiments, the method 2100 can include generating a ROI in the first data after receiving the first data at 2102. In such embodiments, the ROI logic component 1004 can generate the ROI. In some such embodiments, generating the ROI can include applying a machine learning segmentation technique to the first data (received at 2102).

[0054] In some embodiments, the method 2100 can include, after training the ML model at 2106, deploying the ML model (e.g., to a plurality of microscope systems) in accordance with any of the embodiments discussed herein (e.g., as discussed above with reference to the classification logic component 1006 and / or the operations of 2004 of the method 2100).

[0055] As mentioned above, in some embodiments in which an initial classification of a ROI is “unclassified,” the classification logic component 1006 can store the “unclassified” classification of the ROI, and can update the classification of the ROI upon notification that a classification from a central server is available. Figure 6 is a flowchart of a method 2200 of creating new classifications for previously unclassified ROIs in accordance with various embodiments. In Figure 6 The operations in are each illustrated in a particular order, but the operations can be reordered and / or repeated as necessary and as appropriate (e.g., different operations can be performed in parallel where appropriate).

[0056] At 2202, first analysis mode data representing a set of ROIs of a sample can be received. The ROIs associated with the first analysis mode data received at 2202 can have previously been classified by a machine learning model as not corresponding to a known classification (e.g., “unclassified”), and the first analysis mode data can have been generated by a microscope system using a first analysis mode (e.g., BSED). The instrument interface logic component 1002 can perform the operations of 2202. In some embodiments, a single ROI corresponds to a single particle in a physical sample, while in other embodiments, a single ROI can correspond to a different component or region in a sample (e.g., a particular structure or component of a biological or non-biological sample).

[0057] At 2204, the first analysis mode data can be clustered. The classification logic component 1006 can perform the operations of 2204. In some embodiments, the first analysis mode data can be clustered at 2204 based on similarities between the first analysis mode data corresponding to different ROIs (e.g., the more similar the first analysis mode data associated with a first ROI and the first analysis mode data associated with a second ROI, the closer the two sets of clustered first analysis mode data can be). Thus, the present disclosure can relate to clustering of ROIs associated with first analysis mode data.

[0058] At 2206, new first analysis mode data representing a new ROI can be received. Instrument interface logic 1002 can perform the operation of 2206.

[0059] At 2208, a determination can be made that the new first analysis mode data belongs to a particular cluster of the clusters generated at 2204. This determination can be based on a comparison of the new first analysis mode data to the analysis mode data clustered at 2204, where the new first analysis mode data is included in a particular cluster with the most similar previously received first analysis mode data. Classification logic 1006 can perform the operation of 2208.

[0060] At 2210, second analysis mode data representing the new ROI can be received. The second analysis mode data can be generated by the microscope system using a second analysis mode (associated with the data received at 2202) that is different from the first analysis mode. Instrument interface logic 1002 can perform the operation of 2210. In some embodiments, the microscope system takes less time to image the portion of the sample associated with the ROI using the first analysis mode than using the second analysis mode. For example, the first analysis mode can include BSED, and the second analysis mode can include EDS.

[0061] At 2212, an identification of a new classification associated with the particular cluster (to which the new ROI was determined to belong at 2208) can be received, and at 2214, the ROIs in the particular cluster can be reclassified to the new classification. User interface logic 1010 can perform the operation of 2212, and classification logic 1006 can perform the operation of 2214. In some embodiments, the identification received at 2212 can be specified by a user, while in other embodiments, the identification can be automatically generated. For example, in some embodiments, a user can review the first analysis mode data (received at 2202 and 2206) including the ROIs in the particular cluster, and in conjunction with additional information provided by the second analysis mode data (received at 2210), can determine that the ROIs in the particular cluster that were previously “unclassified” should all be assigned a new classification (e.g., a new type of contaminant, a new type of particle, etc.), and the user can provide the new classification to the particular cluster. In some embodiments, user interface logic 1010 can facilitate the operation of 2212 by providing the user with a list of the ROIs in the particular cluster and the second analysis mode data via a GUI (e.g., as discussed below) and the GUI can allow the user to type or otherwise input the new classification that can be assigned to all or some of the ROIs in the particular cluster. In other embodiments, the new classification can be automatically identified by module 1000 based on other data available to module 1000. Figure 7

[0062] ​In some implementations, method 2200 may further include: after reclassifying the ROI at 2212, outputting a classification report that includes the classification of the individual ROIs, including the physical samples. User interface logic component 1010 may perform this operation. The classification report may include the location of the individual ROI, the morphological characteristics of the portion of the physical sample corresponding to the individual ROI, and / or any other available or derived information about the physical samples.

[0063] In some implementations, the methods disclosed herein may be performed by a single microscope system and / or by a central server communicating with multiple microscope systems. For example, in some implementations, the central server may receive classification-related data, train and retrain an ML model to perform classification, and once the performance of the ML model exceeds a threshold, the ML model may be pushed to different microscope systems communicating with the server; the different microscope systems can then use the ML model to perform classification of the components of the analyzed samples. In some implementations, a method 2200 for identifying appropriate classifications of previously unclassified ROIs may be performed by the central server, which may be configured to update the ML model and / or previous classifications with new classifications as soon as they become available.

[0064] As discussed in the user interface logic component 1010 of the sample analysis module 1000 herein, the sample analysis methods disclosed herein may include interaction with human users (e.g., via the reference herein). Figure 9 The discussion focuses on the user's local computing device 5020. These interactions may include providing the user with information (e.g., about the sample being analyzed or by means of, as discussed below). Figure 9 Scientific instruments such as the 5010 perform other tests or measurements using information from local or remote databases or other information, or provide users with input commands (e.g., for controlling things like...). Figure 9 The options for operating scientific instruments such as the scientific instrument 5010, or for controlling the analysis of data generated by the scientific instrument, querying (e.g., querying local or remote databases), or other information are provided. In some implementations, these interactions can be performed via a GUI that includes a display device (e.g., referenced herein). Figure 8 The visual display on the display device 4010 discussed herein (e.g., via reference hereinafter) Figure 8 One or more input devices, such as a keyboard, mouse, touchpad, or touchscreen, included in the other I / O devices discussed 4012, provide output to the user and / or prompt the user for input. The sample analysis system disclosed herein may include any suitable GUI for interacting with the user.

[0065] Figure 7Example GUI 3000s, according to various embodiments, are depicted that can be used to perform some or all of the sample analysis methods disclosed herein. As described above, GUI 3000 can be set up in a sample analysis system (e.g., as referenced herein). Figure 9 The computing device of the sample analysis system 5000 discussed herein (e.g., the referenced in this paper) Figure 8 The display device of the computing device 4000 discussed herein (e.g., the one referred to herein) Figure 8 The display device discussed is 4010, and the user can use any suitable input device (e.g., included in the references herein). Figure 8 The other I / O devices discussed (any of the input devices in 4012) and input technologies (e.g., cursor movement, motion capture, facial recognition, gesture detection, voice recognition, button actuation, etc.) interact with the GUI 3000.

[0066] The GUI 3000 may include a data display area 3002, a data analysis area 3004, a scientific instrument control area 3006, and a settings area 3008. Figure 7 The specific number and arrangement of areas depicted are merely illustrative, and any number and arrangement of areas (including any desired features) may be included in the GUI 3000.

[0067] Data display area 3002 can display data generated by scientific instruments (e.g., as referenced in this article). Figure 9 The data generated by the scientific instrument 5010 discussed. For example, the data display area 3002 may display images or other data (e.g., EDS data) representing physical samples (e.g., data generated by one or more microscope systems and received by the instrument interface logic unit 1002 of the sample analysis module 1000). In some embodiments, one or more ROIs identified by the ROI logic unit 1004 may be displayed on or together with associated image data in the data display area 3002.

[0068] Data analysis area 3004 may display the results of data analysis (e.g., the results of analyzing the data and / or other data illustrated in data display area 3002). For example, data analysis area 3004 may display the morphological characteristics of a calculated individual ROI, the location of the individual ROI, and the classification of the ROI (e.g., as generated by classification logic unit 1006 and evaluated by evaluation logic unit 1008). In some embodiments, data analysis area 3004 may include a classification report as discussed herein, or may include an option for users to download or transmit classification reports. In some embodiments, data display area 3002 and data analysis area 3004 may be combined in GUI 3000 (e.g., to include data output from scientific instruments and some analysis of the data in a public graphic or area).

[0069] The scientific instrument control region 3006 can include options that allow a user to control a scientific instrument (e.g., the scientific instrument 5010 discussed herein with reference to FIG. 5). For example, the scientific instrument control region 3006 can include controls for the instrument interface logic 1002 (e.g., to initiate a second analysis mode for analyzing a portion of a sample, as discussed herein). Figure 9

[0070] The settings region 3008 can include options that allow a user to control features and functions of the GUI 3000 (and / or other GUIs), and / or to perform common computing operations with respect to the data display region 3002 and the data analysis region 3004 (e.g., save data on a storage device, such as the storage device 4004 discussed herein with reference to FIG. 4, transmit data to another user, tag data, etc.). For example, the settings region 3008 can include controls for updating ML models, deploying ML models from a server, or other kinds of controls. Figure 8

[0071] As discussed above, the sample analysis module 1000 can be implemented by one or more computing devices. Figure 8 is a block diagram of a computing device 4000 that can perform some or all of the sample analysis methods disclosed herein, in accordance with various embodiments. In some embodiments, the sample analysis module 1000 can be implemented by a single computing device 4000 or by multiple computing devices 4000. Further, as discussed below, the computing device 4000 (or multiple computing devices 4000) that implements the sample analysis module 1000 can be part of one or more of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040. Figure 9

[0072] Figure 8 The computing device 4000 is illustrated as having a number of components, but any one or more of these components can be omitted or repeated according to the suitability of the application and the setting. In some embodiments, some or all of the components included in the computing device 4000 can be attached to one or more motherboards and enclosed in a housing (e.g., including plastic, metal, and / or other materials). In some embodiments, some of these components can be manufactured onto a single system on a chip (SoC) (e.g., the SoC can include one or more processing devices 4002 and one or more storage devices 4004). Additionally, in various embodiments, the computing device 4000 can not include Figure 8 ​​​One or more of the illustrated components can be included in the computing device 4000, but can include interface circuitry (not shown) for coupling to one or more components using any suitable interface (e.g., a universal serial bus (USB) interface, a high-definition multimedia interface (HDMI) interface, a controller area network (CAN) interface, a serial peripheral interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface). For example, the computing device 4000 can not include the display device 4010, but can include display device interface circuitry (e.g., a connector and driver circuitry) to which the display device 4010 can be coupled.

[0073] The computing device 4000 can include a processing device 4002 (e.g., one or more processing devices). As used herein, the term “processing device” can refer to any device or portion of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory. The processing device 4002 can include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), cryptographic processors (specialized processors that perform cryptographic algorithms within hardware), server processors, or any other suitable processing devices.

[0074] The computing device 4000 can include a storage device 4004 (e.g., one or more storage devices). The storage device 4004 can include one or more memory devices, such as random access memory (RAM) (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive-bridge RAM (CBRAM) devices), hard-disk drive-based memory devices, solid-state memory devices, network drives, cloud drives, or any combination of memory devices. In some embodiments, the storage device 4004 can include memory that shares a die with the processing device 4002. In such embodiments, the memory can function as cache memory and can include, for example, embedded dynamic random access memory (eDRAM) or spin-transfer torque magnetic random access memory (STT-MRAM). In some embodiments, the storage device 4004 can include a non-transitory computer-readable medium having instructions thereon that, when executed by one or more processing devices (e.g., the processing device 4002), cause the computing device 4000 to perform any suitable method disclosed herein, or portions thereof.

[0075] The computing device 4000 can include an interface device 4006 (such as one or more interface devices 4006). The interface device(s) 4006 can include one or more communication chips, connectors, and / or other hardware and software to manage communications between the computing device 4000 and other computing devices. For example, the interface device(s) 4006 can include circuitry to manage wireless communications for the transfer of data to and from the computing device 4000. The terms “wireless” and “wireless ly” can be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that can communicate data through the use of modulated electromagnetic radiation through a non-solid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not. The circuitry included in the interface device(s) 4006 to manage wireless communications can implement any of a number of wireless standards or protocols, including but not limited to IEEE standards including Wi-Fi (the IEEE 802.11 family of standards), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), the Long-Term Evolution (LTE) project

[0076] In some embodiments, the interface device 4006 can include circuitry for managing wired communications, such as electrical communication protocols, optical communication protocols, or any other suitable communication protocols. For example, the interface device 4006 can include circuitry that supports communications in accordance with Ethernet technology. In some embodiments, the interface device 4006 can support both wireless and wired communications, and / or can support multiple wired communication protocols and / or multiple wireless communication protocols. For example, a first set of circuitry of the interface device 4006 can be dedicated to short-range wireless communications such as Wi-Fi or Bluetooth, while a second set of circuitry of the interface device 4006 can be dedicated to long-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, and so on. In some embodiments, a first set of circuitry of the interface device 4006 can be dedicated to wireless communications, while a second set of circuitry of the interface device 4006 can be dedicated to wired communications.

[0077] The computing device 4000 can include a battery / power supply circuit 4008. The battery / power supply circuit 4008 can include one or more energy storage devices (e.g., batteries or capacitors), and / or circuitry for coupling components of the computing device 4000 to an energy source separate from the computing device 4000 (e.g., an AC line power supply).

[0078] The computing device 4000 can include a display device 4010 (e.g., a plurality of display devices). The display device 4010 can include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0079] The computing device 4000 can include other input / output (I / O) devices 4012. The other I / O devices 4012 can include, for example, one or more audio output devices (e.g., speakers, headphones, earbuds, an alarm, etc.), one or more audio input devices (e.g., microphones or microphone arrays), a positioning device (e.g., a GPS device that communicates with satellite-based systems to receive a location of the computing device 4000 as known in the art), an audio codec, a video codec, a printer, a sensor (e.g., a thermocouple or other temperature sensor, a humidity sensor, a pressure sensor, a vibration sensor, an accelerometer, a gyroscope, etc.), an image capture device such as a camera, a keyboard, a cursor control device such as a mouse, a stylus, a trackball, or a touchpad, a barcode reader, a quick response (QR) code reader, or a radio frequency identification (RFID) reader.

[0080] The computing device 4000 can have any suitable form factor for its application and setting, such as a handheld or mobile computing device (e.g., a cellular phone, a smartphone, a mobile Internet device, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultra-mobile personal computer, etc.), a desktop computing device, or a server computing device or other networked computing component.

[0081] One or more computing devices implementing any of the sample analysis modules or methods disclosed herein can be part of a sample analysis system. Figure 9 is a block diagram of an example sample analysis system 5000 in which some or all of the sample analysis methods disclosed herein can be performed in accordance with various embodiments. The sample analysis modules and methods disclosed herein (e.g., Figure 1 the sample analysis module 1000 and Figure 4 the method 2000 of FIG. 2) can be implemented by one or more of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 of the sample analysis system 5000.

[0082] Any of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 can include any of the embodiments of the computing device 4000 discussed herein with reference to Figure 8 Any of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 can take the form of any appropriate embodiment of the computing device 4000 discussed herein with reference to Figure 8 Any of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 can take the form of any appropriate embodiment of the computing device 4000 discussed herein with reference to

[0083] The scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 can each include a processing device 5002, a storage device 5004, and an interface device 5006. The processing device 5002 can take the form of any appropriate form, including any of the processing devices 4002 discussed herein with reference to Figure 8 The processing device 5002 included in different ones of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 can take the same form or different forms. The storage device 5004 can take any suitable form, including any of the storage devices 4004 discussed herein with reference to Figure 8The form of any of the storage devices 4004 discussed, and the storage devices 5004 included in different devices of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 can take the same form or different forms. The interface devices 5006 can take any suitable form, including the forms discussed herein with reference to the interface devices 4006 of the computing device 4000. Figure 8 The form of any of the interface devices 4006 discussed, and the interface devices 5006 included in different devices of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 can take the same form or different forms.

[0084] The scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, and the remote computing device 5040 can communicate with other elements of the sample analysis system 5000 via communication paths 5008. The communication paths 5008 can communicatively couple the interface devices 5006 of different ones of the elements of the sample analysis system 5000, as shown, and can be wired or wireless communication paths (e.g., according to any of the communication technologies discussed herein with reference to the interface devices 4006 of the computing device 4000). Figure 8 Figure 9 The particular sample analysis system 5000 depicted in FIG. 5 includes a communication path between each pair of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, and the remote computing device 5040, but such a “fully connected” implementation is merely illustrative, and in various implementations, various ones of the communication paths 5008 can not be present. For example, in some implementations, the service local computing device 5030 can not have a direct communication path 5008 between its interface device 5006 and the interface device 5006 of the scientific instrument 5010, but can communicate with the scientific instrument 5010 via the communication path 5008 between the service local computing device 5030 and the user local computing device 5020 and the communication path 5008 between the user local computing device 5020 and the scientific instrument 5010.

[0085] The scientific instrument 5010 can include any appropriate scientific instrument, such as a charged particle microscope (e.g., an electron microscope), an optical microscope, a spectroscopy device, or any other suitable analysis instrument.

[0086] ​A user local computing device 5020 can be a computing device local to a user of the scientific instrument 5010 (e.g., according to any of the embodiments of the computing device 4000 discussed herein). In some embodiments, the user local computing device 5020 can also be located locally to the scientific instrument 5010, although this is not a requirement; for example, a user local computing device 5020 in a user’s home or office can be remote from the scientific instrument 5010, but in communication with the scientific instrument 5010 so that the user can use the user local computing device 5020 to control and / or access data from the scientific instrument 5010. In some embodiments, the user local computing device 5020 can be a laptop, a smartphone, or a tablet device. In some embodiments, the user local computing device 5020 can be a portable computing device.

[0087] A service local computing device 5030 can be a computing device local to an entity that services the scientific instrument 5010 (e.g., according to any of the embodiments of the computing device 4000 discussed herein). For example, the service local computing device 5030 can be a local device of a manufacturer of the scientific instrument 5010 or a third-party service company. In some embodiments, the service local computing device 5030 can be in communication with the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., via the direct communication path 5008 or via a plurality of “indirect” communication paths 5008, as described above) to receive data regarding the operation of the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., self-test results of the scientific instrument 5010, calibration coefficients used by the scientific instrument 5010, measurements of sensors associated with the scientific instrument 5010, etc.). In some embodiments, the service local computing device 5030 can be in communication with the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., via the direct communication path 5008 or via a plurality of “indirect” communication paths 5008, as described above) to send data to the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., to update programmed instructions (such as firmware) in the scientific instrument 5010 to initiate performance of a test or calibration sequence in the scientific instrument 5010, to update programmed instructions (such as software) in the user local computing device 5020 or the remote computing device 5040, etc.). A user of the scientific instrument 5010 can utilize the scientific instrument 5010 or the user local computing device 5020 to communicate with the service local computing device 5030 to report a problem with the scientific instrument 5010 or the user local computing device 5020, to request a technician visit to improve operation of the scientific instrument 5010, to order consumables or replacement parts associated with the scientific instrument 5010, or for other purposes.

[0088] Remote computing device 5040 may be a computing device located remotely from scientific instrument 5010 and / or user local computing device 5020 (e.g., any implementation of the computing device 4000 discussed herein). In some implementations, remote computing device 5040 may be included in a data center or other large-scale server environment. In some implementations, remote computing device 5040 may include network-attached storage (e.g., as part of storage device 5004). Remote computing device 5040 may store data generated by scientific instrument 5010, perform analysis on the data generated by scientific instrument 5010 (e.g., according to programmed instructions), facilitate communication between user local computing device 5020 and scientific instrument 5010, and / or facilitate communication between service local computing device 5030 and scientific instrument 5010. In some implementations, remote computing device may be a server that manages and deploys ML models and implements new classifications of previously unclassified ROIs, as discussed herein.

[0089] In some implementation schemes, it may not exist. Figure 9 One or more components of the illustrated sample analysis system 5000. Additionally, in some embodiments, there may be... Figure 9The sample analysis system 5000 can include multiple elements of various ones of the elements of the sample analysis system 5000. For example, the sample analysis system 5000 can include multiple user local computing devices 5020 (e.g., different user local computing devices 5020 associated with different users or at different locations). In another example, the sample analysis system 5000 can include multiple scientific instruments 5010 all in communication with the service local computing device 5030 and / or the remote computing device 5040; in such an implementation, the service local computing device 5030 can monitor these multiple scientific instruments 5010, and the service local computing device 5030 can cause updates or other information to be “broadcast” to the multiple scientific instruments 5010 at the same time. Different ones of the scientific instruments 5010 in the sample analysis system 5000 can be located close to one another (e.g., in the same room) or far from one another (e.g., on different floors of a building, in different buildings, in different cities, etc.). In some implementations, the scientific instruments 5010 can be connected to an Internet of Things (IoT) stack that allows the scientific instruments 5010 to be commanded and controlled through web-based applications, virtual or augmented reality applications, mobile applications, and / or desktop applications. Any of these applications can be accessible by a user operating a user local computing device 5020 that communicates with the scientific instruments 5010 through an intermediary remote computing device 5040. In some implementations, the scientific instruments 5010 can be sold by a manufacturer along with one or more associated user local computing devices 5020 as part of a local scientific instrument computing unit 5012.

[0090] The following paragraphs provide various examples of the implementations disclosed herein.

[0091] Example A includes any of the sample analysis modules disclosed herein.

[0092] Example B includes any of the methods disclosed herein.

[0093] Example C includes any of the GUIs disclosed herein.

[0094] Example D includes any of the sample analysis computing devices and systems disclosed herein.

[0095] Example 1 is a method for classifying microscopic components of a physical sample, the method comprising: generating a set of regions of interest (ROIs) in an image representing the physical sample, wherein the image is generated by a microscopy system using a first analysis mode; generating an initial classification for a ROI by applying a trained machine learning model to at least a portion of the image associated with the ROI; generating a confidence score associated with the initial classification; and causing the microscopy system to reanalyze at least a portion of the sample associated with the ROI using a second analysis mode different from the first analysis mode when the confidence score of the initial classification of a ROI does not satisfy a set of confidence criteria.

[0096] Example 2 includes the subject matter of Example 1, and further specifies that the microscopy system takes less time to image the portion of the sample associated with the ROI using the first analysis mode than using the second analysis mode.

[0097] Example 3 includes the subject matter of any of Examples 1-2, and further specifies that the first analysis mode comprises backscattered electron detection (BSED).

[0098] Example 4 includes the subject matter of Example 3, and further specifies that the second analysis mode comprises energy dispersive spectroscopy (EDS).

[0099] Example 5 includes the subject matter of any of Examples 1-4, and further specifies that some, but not all, of the ROIs are reanalyzed using the second analysis mode.

[0100] Example 6 includes the subject matter of any of Examples 1-5, and further specifies that generating the set of ROIs comprises applying a machine learning segmentation technique to the image.

[0101] Example 7 includes the subject matter of any of Examples 1-6, and further specifies that a single ROI in the image corresponds to a single particle in the physical sample.

[0102] Example 8 includes the subject matter of Example 7, and further specifies that the initial classification for a ROI is selected from a set of at least two classifications, and the set of at least two classifications includes gunshot residue (GSR) but not GSR.

[0103] Example 9 includes the subject matter of Example 7, and further specifies that the initial classification for a ROI is selected from a set of at least two classifications, and the set of at least two classifications includes worn and soft.

[0104] Example 10 includes the subject matter of any one of Examples 1-9, and further comprises generating a final classification of the ROI using data generated by the reanalysis when the confidence score of the initial classification of the ROI does not satisfy the set of confidence criteria.

[0105] Example 11 includes the subject matter of Example 10, and further specifies that the data generated by the reanalysis comprises elemental composition data.

[0106] Example 12 includes the subject matter of any one of Examples 10-11, and further comprises using the initial classification of the ROI as a final classification of the ROI when the confidence score of the initial classification of the ROI does not satisfy a set of confidence criteria.

[0107] Example 13 includes the subject matter of Example 12, and further comprising outputting a classification report comprising a final classification of individual ROIs of a set of ROIs.

[0108] Example 14 includes the subject matter of Example 13, and further specifies that the classification report comprises a location of an individual ROI.

[0109] Example 15 includes the subject matter of any one of Examples 13-14, and further specifies that the classification report comprises a morphological property of the portion of the sample corresponding to an individual ROI.

[0110] Example 16 includes the subject matter of Example 15, and further specifies that the morphological property comprises an area, a circularity, or a roughness.

[0111] Example 17 includes the subject matter of any one of Examples 1-16, and further comprising receiving the trained machine learning model from a central server prior to generating an initial classification of a ROI by applying the trained machine learning model.

[0112] Example 18 includes the subject matter of Example 17, and further specifies that the central server deploys the machine learning model to a plurality of microscope systems.

[0113] Example 19 includes the subject matter of any one of Examples 1-18, and further specifies that the initial classification of a ROI is selected from a set of at least two classifications, and the set of at least two classifications comprises unclassified.

[0114] Example 20 includes the subject matter of Example 19, and further comprising receiving a final classification of a ROI upon notification from a central server that a classification is available when the initial classification of the ROI is unclassified.

[0115] Example 21 is a method for generating a machine learning model for classifying microscopic components of a physical sample, the method comprising: receiving first data representing the physical sample, wherein the first data is generated by a microscope system using a first analysis mode; receiving classification data representing a classification of a region of interest (ROI) in the first data, wherein the classification data is generated based at least in part on second data of the physical sample different from the first data, wherein the second data is generated by the microscope system using a second analysis mode different from the first analysis mode; training a machine learning model using data representing the first analysis data and the classification data without using the second data to generate a classification of a ROI in data generated by the microscope system using the first analysis mode.

[0116] Example 22 includes the subject matter of Example 21, and further including: after training the machine learning model, generating a training performance score.

[0117] Example 23 includes the subject matter of Example 22, and further including: when the training performance score does not satisfy a training performance criterion, retraining the machine learning model based on additional first data and additional corresponding classification data.

[0118] Example 24 includes the subject matter of Example 22, and further including: when the training performance score satisfies a training performance criterion, providing the machine learning model to generate a classification of a ROI in an additional image of a physical sample, wherein the additional image is generated by the microscope system using the first analysis mode.

[0119] Example 25 includes the subject matter of any of Examples 21-24, and further specifying that the microscope system takes less time to image a portion of a sample using the first analysis mode than using the second analysis mode.

[0120] Example 26 includes the subject matter of any of Examples 21-25, and further specifying that the first analysis mode comprises backscattered electron detection (BSED).

[0121] Example 27 includes the subject matter of Example 26, and further specifying that the second analysis mode comprises energy dispersive spectroscopy (EDS).

[0122] Example 28 includes the subject matter of any of Examples 21-27, and further including: generating the ROI in the first data.

[0123] Example 29 includes the subject matter of Example 28, and further specifies that generating the ROI includes applying a machine learning segmentation technique to the first data.

[0124] Example 30 includes the subject matter of any of Examples 21-29, and further specifies that a single ROI in the first data corresponds to a single particle in the physical sample.

[0125] Example 31 includes the subject matter of any of Examples 21-30, and further specifies that the classification generated by the machine learning model is selected from a set of at least two classifications, and the set of at least two classifications includes gunshot residue (GSR) but not GSR.

[0126] Example 32 includes the subject matter of any of Examples 21-30, and further specifies that the classification generated by the machine learning model is selected from a set of at least two classifications, and the set of at least two classifications includes worn and soft.

[0127] Example 33 includes the subject matter of any of Examples 21-32, and further specifies that the second data includes elemental composition data.

[0128] Example 34 includes the subject matter of any of Examples 21-33, and further specifies that training the machine learning model using data representative of the first analysis data includes training the machine learning model using the first analysis data.

[0129] Example 35 includes the subject matter of any of Examples 21-33, and further specifies that training the machine learning model using data representative of the first analysis data includes training the machine learning model using a morphology parameter generated at least in part from the first analysis data.

[0130] Example 36 includes the subject matter of Example 35, and further specifies that the morphology parameter includes area, circularity, or roughness.

[0131] Example 37 includes the subject matter of any of Examples 21-36, and further specifies that the machine learning model is configured to receive a two-dimensional image as input.

[0132] Example 38 includes the subject matter of any of Examples 21-36, and further specifies that the machine learning model is configured to receive a one-dimensional vector as input.

[0133] Example 39 includes the subject matter of Example 38, and further specifies that the one-dimensional vector includes at least some morphology parameters.

[0134] Example 40 includes the subject matter of any one of Examples 21-39, and further comprises deploying the machine learning model to a plurality of microscope systems.

[0135] Example 41 is a method for classifying microcomponents of a physical sample, the method comprising: receiving first analytical mode data representing a set of regions of interest (ROIs) in an image of a physical sample, wherein the ROIs were previously classified by a machine learning model as not corresponding to a known classification, and wherein the first analytical mode data was generated by a microscope system using a first analytical mode; clustering the first analytical mode data based on similarity between the first analytical mode data corresponding to different ROIs; receiving new first analytical mode data representing a new ROI; determining that the new first analytical mode data indicates that the new ROI belongs to a particular cluster; receiving second analytical mode data representing the new ROI, wherein the second analytical mode data was generated by the microscope system using a second analytical mode different from the first analytical mode; providing a list of ROIs in the particular cluster and the second analytical mode data to a user; receiving an identification of a new classification associated with the particular cluster; and reclassifying the ROIs in the particular cluster to the new classification.

[0136] Example 42 includes the subject matter of Example 41, and further specifies that the microscope system takes less time to image the portion of the sample associated with the ROI using the first analytical mode than using the second analytical mode.

[0137] Example 43 includes the subject matter of any one of Examples 41-42, and further specifies that the first analytical mode comprises backscattered electron detection (BSED).

[0138] Example 44 includes the subject matter of Example 43, and further specifies that the second analytical mode comprises energy dispersive spectroscopy (EDS).

[0139] Example 45 includes the subject matter of any one of Examples 41-44, and further specifies that a single ROI corresponds to a single particle in the physical sample.

[0140] Example 46 includes the subject matter of any one of Examples 41-45, and further comprises, after reclassifying the ROIs, outputting a classification report comprising classifications of single ROIs of the physical sample.

[0141] Example 47 includes the subject matter of Example 46, and further specifies that the classification report comprises locations of single ROIs.

[0142] Example 48 includes the subject matter of any one of Examples 46-47, and further specifies that the classification report includes a morphological characteristic of a portion of the physical sample corresponding to a single ROI.

[0143] Example 49 includes the subject matter of any one of Examples 41-48, and further specifies that the microscope system is a charged particle microscope system.

[0144] Example 50 includes the subject matter of any one of Examples 41-49, and further specifies that the physical sample is a non-biological sample.

[0145] Example 51 includes the subject matter of any one of Examples 41-49, and further specifies that receiving the identification of the new classification includes receiving a user specification of the new classification.

[0146] Example 52 includes the subject matter of Example 51, and further specifies that the user specification is received through a graphical user interface.

Claims

1. A method of classifying microscopic components of a physical sample, the method comprising: generating a set of regions of interest (ROIs) in an image representing the physical sample, wherein the image is generated by a microscope system using a first analysis mode; generating an initial classification for a ROI by applying a trained machine learning model to at least a portion of the image associated with the ROI; generating a confidence score associated with the initial classification of a ROI; and causing the microscope system to reanalyze at least a portion of the sample associated with the ROI using a second analysis mode different from the first analysis mode when the confidence score of the initial classification of a ROI does not satisfy a set of confidence criteria.

2. The method of claim 1, wherein the microscope system takes less time to image the portion of the sample associated with the ROI using the first analysis mode than using the second analysis mode.

3. The method of claim 1, wherein the first analysis mode comprises backscattered electron detection (BSED).

4. The method of claim 3, wherein the second analysis mode comprises energy dispersive spectroscopy (EDS).

5. The method of claim 1, wherein some, but not all, of the ROIs are reanalyzed using the second analysis mode.

6. The method of claim 1, wherein a single ROI in the image corresponds to a single particle in the physical sample.

7. The method of claim 1, further comprising: generating a final classification of a ROI using data generated by the reanalysis when the confidence score of the initial classification of a ROI does not satisfy the set of confidence criteria.

8. The method of claim 7, further comprising: using the initial classification of a ROI as the final classification of the ROI when the confidence score of the initial classification of a ROI satisfies a set of confidence criteria.

9. The method of claim 8, further comprising: outputting a classification report comprising the final classification of individual ROIs in the set of ROIs.

10. A method for generating a machine learning model for classifying microscopic components of a physical sample, the method comprising: receiving first data representing the physical sample, wherein the first data is generated by a microscope system using a first analysis mode; receiving classification data representing a classification of a region of interest (ROI) in the first data, wherein the classification data is generated based at least in part on second data of the physical sample different from the first data, wherein the second data is generated by the microscope system using a second analysis mode different from the first analysis mode; training a machine learning model using data representing first analysis data and the classification data without using the second data to generate a classification of a ROI in data generated by the microscope system using the first analysis mode.

11. The method of claim 10, further comprising: ​ After training the machine learning model, a training performance score is generated.

12. The method of claim 11, further comprising: retraining the machine learning model based on additional first data and additional corresponding classification data when the training performance score does not satisfy a training performance criterion.

13. The method of claim 11, further comprising: providing the machine learning model to generate classifications of ROIs in additional images of physical samples when the training performance score satisfies a training performance criterion, wherein the additional images are generated by the microscope system using the first analysis mode.

14. The method of claim 10, wherein training the machine learning model using data representative of the first analysis data comprises training the machine learning model using the first analysis data.

15. The method of claim 10, wherein training the machine learning model using data representative of the first analysis data comprises training the machine learning model using morphological parameters generated at least in part from the first analysis data.

16. The method of claim 10, further comprising: deploying the machine learning model to a plurality of microscope systems.

17. A method of classifying microcomponents of a physical sample, the method comprising: receiving first analysis mode data representative of a set of regions of interest (ROIs) in an image of a physical sample, wherein the ROIs were previously classified by a machine learning model as not corresponding to a known classification, and wherein the first analysis mode data was generated by a microscope system using a first analysis mode; clustering the first analysis mode data based on similarity between the first analysis mode data corresponding to different ROIs; receiving new first analysis mode data representative of a new ROI; determining that the new first analysis mode data indicates that the new ROI belongs to a particular cluster; receiving second analysis mode data representative of the new ROI, wherein the second analysis mode data was generated by a microscope system using a second analysis mode different from the first analysis mode; providing a list of ROIs in the particular cluster and the second analysis mode data to a user; receiving an identification of a new classification associated with the particular cluster; and reclassifying the ROIs in the particular cluster as the new classification.

18. The method of claim 17, wherein the physical sample is a non-biological sample.

19. The method of claim 17, wherein receiving the identification of the new classification comprises receiving a user specification of the new classification.

20. The method of claim 17, wherein the user specification is received through a graphical user interface. ​

Images