Classification of microscopic components of physical sample

Abstract

To provide a sample analysis method capable of accurately classifying the microscopic components of a sample much more rapidly than conventional classification systems.SOLUTION: A method for classifying microscopic components of a physical sample includes generating a set of regions of interest (ROIs) within an image representing the physical sample. The image is generated by a microscope system using a first analysis mode. The method further includes applying a trained machine learning model to at least a portion of the image associated with the ROIs to generate an initial classification of the ROIs, generating a confidence score associated with the initial classification, and, when the confidence score for the initial classification of an ROI does not meet a set of confidence criteria, causing the microscope system to re-analyze at least a portion of the sample associated with the ROI using a second analysis mode different from the first analysis mode.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] Many types of analytical instruments exist that can generate data about the microscopic characteristics of a sample. Different types of such instruments can use different physical principles to generate data about the sample. [Brief explanation of the drawings]

[0002]

[0013] The embodiments will be readily understood by the following detailed description taken in conjunction with the accompanying drawings, in which:

[0014] To facilitate this description, like reference numerals refer to like structural elements;

[0015] The embodiments are illustrated in the figures of the accompanying drawings, by way of example, and not by way of limitation. [Figure 1] FIG. 1 is a block diagram of an exemplary sample analysis module for performing sample analysis operations, according to various embodiments. [Figure 2] 1 illustrates an example of an image in which multiple regions-of-interest (ROIs) have been identified, according to various embodiments. [Figure 3] 1 illustrates exemplary results of an energy dispersive spectroscopy (EDS) analysis on a portion of a sample associated with a particular ROI, according to various embodiments. [Figure 4] FIG. 1 is a flow diagram of an exemplary method for performing a sample analysis operation, according to various embodiments. [Figure 5] FIG. 1 is a flow diagram of an exemplary method for generating a machine learning model for classifying microscopic components of a physical sample, according to various embodiments. [Figure 6] FIG. 1 is a flow diagram of a method for creating a new classification for a previously unclassified ROI, according to various embodiments. [Figure 7] 1 is an example of a graphical user interface (GUI) that may be used in implementing some or all of the sample analysis methods disclosed herein, according to various embodiments. [Figure 8] FIG. 1 is a block diagram of an exemplary computing device that may implement some or all of the sample analysis methods disclosed herein, according to various embodiments. [Figure 9] FIG. 1 is a block diagram of an exemplary sample analysis system in which some or all of the sample analysis methods disclosed herein may be implemented, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0003]

[0006] Disclosed herein are systems for classifying microscopic components of a physical sample, as well as related methods, computing devices, and computer-readable media. For example, in some embodiments, a method for classifying microscopic components of a physical sample may include generating a set of regions of interest (ROIs) in an image representing the physical sample, the image being generated by a microscope system using a first analysis mode, generating an initial classification of the ROIs by applying a trained machine-learning (ML) model to at least a portion of the image associated with the ROIs, generating a confidence score associated with the initial classification, and causing the microscope system to re-analyze at least a portion of the sample associated with the ROIs using a second analysis mode different from the first analysis mode when the confidence score for the initial classification of the ROIs does not satisfy a set of confidence criteria.

[0004] As discussed in more detail below, many scientific and industrial applications can benefit from accurate classification of a sample's microscopic components (e.g., to avoid using "dirty" components in an assembly process, to ensure materials have the appropriate quality before further processing, to facilitate effective law enforcement by rapid analysis of crime scene samples, etc.). In many such applications, generating an accurate classification as quickly as possible is critical to enable the timely use of this information. The need for speed is particularly important in high-volume analytical applications where there are many samples requiring analysis (e.g., every tenth part in some automotive manufacturing processes), and longer processing times can result in undesirable backlogs. In some applications, the analysis must be non-destructive to the sample, highly repeatable (e.g., 93-99% of particles detected and matched), and / or reproducible when different microscope systems are used (e.g., so that classification results from imaging a sample using one microscope match classification results from imaging a sample using a different microscope; some applications specify only a tolerance of + / - 8% or less for error as part of field acceptance testing).

[0005] Disclosed herein are sample analysis techniques and systems that can enable accurate classification of a sample's microscopic components much more quickly than traditional classification systems. The techniques and systems disclosed herein can utilize machine learning techniques and can be implemented without requiring a human user to set aside additional time to carefully craft a set of training data (as is typically required for ML tools). Furthermore, various embodiments of the techniques and systems disclosed herein provide an autonomous or semi-autonomous process for the development of ML-based classifiers, enabling model training and deployment with little or no human effort. Once deployed, the ML-based techniques and systems disclosed herein can enable classification of a sample's components much faster than traditional classification systems, with the potential for orders of magnitude improvement in sample evaluation time.

[0006] Thus, sample analysis embodiments disclosed herein can achieve improved performance compared to conventional approaches. In particular, therefore, the embodiments disclosed herein provide improvements to scientific instrument technology (e.g., improvements to the computer technology supporting such scientific instrumentation, among other improvements). Various of the embodiments disclosed herein can improve upon conventional approaches to achieve the technical advantage of faster classification of microscopic components of physical samples through autonomous or semi-autonomous training and deployment of ML models to at least partially replace time-consuming conventional analytical techniques and achieve higher throughput. Such technical advantages are not achievable through routine and conventional approaches, and all users of systems incorporating such embodiments can benefit from these advantages. Thus, the technical features of the embodiments disclosed herein, as well as the combination of features of the embodiments disclosed herein, are clearly unconventional in the field of sample analysis. As discussed further herein, various aspects of the embodiments disclosed herein can improve the functionality of the computer itself, e.g., a computing system that analyzes sample analysis data. The computational and user interface features disclosed herein not only involve the gathering and comparison of information, but also apply new analytical and technological techniques to modify the operation of systems that employ microscopic feature classification as part of a scientific or industrial process. Thus, the present disclosure introduces capabilities that could not be performed by conventional computing devices or humans.

[0007] Thus, embodiments of the present disclosure may serve any of several technical purposes, such as controlling a particular technology system or process, determining from measurements (e.g., classification of the microscopic components of a sample) how to control a machine or process, digital image enhancement or analysis, and providing faster processing of analytical instrument data.

[0008] Accordingly, the embodiments disclosed herein provide improvements in analytical instruments and sample analysis techniques (eg, improvements in computer technology supporting sample analysis, among other improvements).

[0009] In the following detailed description, reference is made to the accompanying drawings that form a part hereof, where like numerals refer to like parts throughout and which show, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description is not to be taken in a limiting sense.

[0010] Various operations may be described as multiple separate actions or operations, in the order most helpful in understanding the subject matter disclosed herein. However, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations may not be performed in the order presented. The operations described may be performed in a different order than in the described embodiment. Various additional operations may be performed and / or described operations may be omitted in additional embodiments.

[0011] For purposes of this disclosure, the phrases “A and / or B” and “A or B” mean (A), (B), or (A and B). For purposes of this disclosure, the phrases “A, B, and / or C” and “A, B, or C” mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Although some elements may be referred to in the singular (e.g., “processing device”), any suitable element may be represented by multiple instances of that element, and vice versa. For example, a set of operations described as being performed by a processing device may be implemented with different ones of the operations being performed by different processing devices. As used herein, the phrase “based on” should be understood to mean “based at least in part on,” unless otherwise specified.

[0012] This description uses the phrases "one embodiment," "various embodiments," and "some embodiments," each of which may refer to one or more of the same or different embodiments. Furthermore, terms such as "comprising," "including," and "having," when used with respect to embodiments of the present disclosure, are synonymous. When used to describe a range of dimensions, the phrase "between X and Y" represents a range that includes X and Y. As used herein, an "apparatus" may refer to any individual device, a collection of devices, a portion of a device, or a collection of portions of devices. The drawings are not necessarily drawn to scale.

[0013] 1 is a block diagram of a sample analysis module 1000 for performing sample analysis operations, according to various embodiments. The sample analysis module 1000 may be implemented by circuitry (e.g., including electrical and / or optical components) such as a programmed computing device. The logic of the sample analysis module 1000 may be contained on a single computing device or may be distributed across multiple computing devices that communicate with each other as needed. An example of a computing device that may implement the sample analysis module 1000, alone or in combination, is discussed herein with reference to the computing device 4000 of FIG. 8, and an example of a system of interconnected computing devices in which the sample analysis module 1000 may be implemented across one or more of the computing devices is discussed herein with reference to the sample analysis system 5000 of FIG. 9.

[0014] The sample analysis module 1000 may include instrument interface logic 1002, region of interest (ROI) logic 1004, classification logic 1006, evaluation logic 1008, and user interface logic 1010. As used herein, the term "logic" may include an apparatus that performs a set of operations associated with the logic. For example, any of the logic elements included within the sample analysis module 1000 may be implemented by one or more computing devices programmed with instructions that cause one or more processing devices of the computing devices to perform the associated set of operations. In particular embodiments, a logic element may include one or more non-transitory computer-readable media having instructions that, when executed by one or more processing devices of the one or more computing devices, cause the one or more computing devices to perform the associated set of operations. As used herein, the term "module" may refer to a collection of one or more logic elements that together perform a function associated with the module. Different logic elements within a module may take the same form or different forms. For example, some logic in a module may be implemented by a programmed general-purpose processing device, while other logic in the module may be implemented by an application-specific integrated circuit (ASIC). In another example, different ones of the logic elements in a module may be associated with different sets of instructions executed by one or more processing devices. A module may not include all of the logic elements depicted in an associated figure; for example, a module may include a subset of the logic elements depicted in an associated figure when the module performs a subset of the operations discussed herein with reference to that module.

[0015] The instrument interface logic 1002 may enable the sample analysis module 1000 to communicate with one or more scientific instruments, such as one or more microscopes (e.g., charged particle microscopes). In some embodiments, this communication may include receiving data from the one or more scientific instruments (e.g., data collected using one or more analysis modes of the scientific instruments) and / or providing commands or instructions to the one or more scientific instruments (e.g., causing the microscope to image the sample over a specified ROI using a specified analysis mode). In various of the embodiments disclosed herein, the scientific instruments that generate the data that will be analyzed by the sample analysis module 1000 may have multiple possible acquisition modes. These modes may generate different types of data, and selecting one or another mode for analysis may require balancing competing factors. For example, some acquisition modes may produce images with higher information density than other modes, but at the cost of longer acquisition times, greater power requirements, a greater risk of damage to the sample, or other factors. Some microscope systems may be configured to operate in, for example, backscattered electron detection (BSED) analysis mode, as well as energy dispersive spectroscopy (EDS) analysis mode; compared to BSED, EDS can provide additional information about the elemental composition of the sample, but may require more time (e.g., 1000 times more to image the same area).

[0016] Although BSED and EDS may be used herein as an example of a pair of analysis types that provide different types or amounts of information about a sample and where one analysis type is more resource intensive than the other, there are many other pairs of analysis types to which the innovative systems and methods disclosed herein may be applied. For example, a first, less resource-intensive analysis type may include any of secondary electron detector (SED) analysis, low vacuum detector (LVD) analysis, circular backscatter detector (CBS) analysis, Everhart-Thornley detector (ETD) analysis, BSED analysis, specific detector analysis (e.g., from in-lens detectors T1 or T2 and / or in-column detector T3), visible light analysis, infrared light analysis, or ultraviolet light analysis, and a second, more resource-intensive analysis type may include any of EDS analysis, electron backscatter diffraction (EBSD) analysis, electron energy loss spectroscopy (EELS) analysis, cathode luminescence (CL) analysis, or wavelength-dispersive x-ray spectroscopy (WDS) analysis.

[0017] The ROI logic 1004 can identify one or more ROIs within an image of a physical sample. The image may be constructed from data generated by a scientific instrument (e.g., a charged particle microscope) and may represent the use of one or more analysis modes by the scientific instrument. As used herein, "image" includes two-dimensional data representations and suitable higher-dimensional data representations. For example, a multi-channel image may include multiple two-dimensional images, one corresponding to each channel (e.g., three two-dimensional images for a red-green-blue (RGB) image capture device). In another example, if an image is generated by data from multiple detectors (e.g., a BSED and a SED), the resulting image may be a three-dimensional matrix (e.g., represented by (w, h, x) coordinates, where w is width, h is height, and x is a vector of data corresponding to different detectors). The ROI can identify a specific portion of the image that corresponds to a feature of interest of the physical sample; the specific feature that is a feature of interest for a particular sample depends on the nature of the sample and the purpose of the sample analysis. In some embodiments, individual ROIs can correspond to individual particles in the physical sample. In some embodiments, an individual ROI may correspond to a cluster of particles in a physical sample (e.g., a stringer in a steel sample), hi some embodiments, an individual ROI may correspond to a structure of interest in a biological sample (e.g., a cell nucleus).

[0018] The ROI logic 1004 may use any suitable technique to identify the ROI within the image. In some embodiments, the ROI logic 1004 may apply conventional segmentation techniques, such as machine learning (ML)-based segmentation techniques or any other computer vision-based techniques (e.g., thresholding, edge detection, etc.) as known in the art, to identify the ROI. In some embodiments where the ROI logic 1004 determines the ROI within a BSED image, the ROI logic 1004 may use BSED thresholding techniques, as known in the art, that identify the ROI using pixels whose intensities fall within a certain range.

[0019] FIG. 2 shows an example of an image 1100 in which multiple ROIs 1102 have been identified, as indicated by the highlighted regions. The image 1100 may represent only a portion of a sample or may be one of many images 1100 tiled or otherwise arranged to represent a field of view. Only some of the ROIs 1102 identified in the image 1100 are labeled in FIG. 2. The image 1100 (which may be, for example, a BSED image) may be generated or received by the instrument interface logic 1002, and the ROIs 1102 may be identified by the ROI logic 1004. FIG. 3 shows example results 1104 of an EDS analysis of a portion of a sample associated with a particular ROI 1102 identified in the image 1100. The results 1104 may include peaks 1106 corresponding to different elements present in the portion of the sample.

[0020] The classification logic 1006 can store one or more ML models that can be used to generate a classification for a particular ROI identified by the ROI logic 1004. The classification logic 1006 can apply the ML model to the ROI to generate such a classification. The classification logic 1006 can generate a new ML model based on a previously performed classification (e.g., manually or using a rule-based process). An exemplary method for creating a new ML model for classification of an ROI in an image of a physical sample is described below with reference to FIG. 5. In some embodiments, the classification logic 1006 can retrain or otherwise update a stored ML model (e.g., based on additional available data). In some embodiments, the classification logic 1006 can deploy one or more ML models to computing devices connected to or included in the scientific instrument, so that the ML models can be executed on those computing devices against data generated by the associated scientific instrument. In some embodiments, the ML models applied by the classification logic 1006 may be received by the classification logic 1006 from a central server (e.g., remote computing device 5040 in FIG. 9 , described below) that may be configured to deploy the ML models to multiple microscope systems. In other embodiments, the classification logic 1006 itself may deploy one or more ML models to other microscope systems.

[0021] The set of possible classifications (e.g., two or more) that may be generated by the ML model of classification logic 1006 may depend on the particular application (e.g., law enforcement, automotive, battery manufacturing, steel processing, etc.) and how the ML model was trained. For example, in some law enforcement applications where materials are analyzed, the set of possible classifications may include gunshot residue (GSR) and may not include GSR. In some automotive applications where the cleanliness of parts may be assessed before assembly, the set of possible classifications may include abrasive (a characterization that may include particles of silicon carbide and other materials) and soft (a characterization that may include aluminum and other materials).

[0022] The architecture of the ML model used by the classification logic 1006 may take any of several forms. In some embodiments, the ML model may take the form of a classifier model known in the art, such as a fully connected neural network. The input to the ML model may be a two-dimensional image or a one-dimensional vector. In some embodiments, the two-dimensional image provided as input to the ML model may be a BSED or other image output by a first analysis mode of the ROI. In some embodiments, the one-dimensional vector provided as input to the ML model may include a set of morphological parameters of the ROI generated based on the results of the first analysis mode. For example, a binary mask may be applied to the BSED or other image to isolate the ROI, and then the morphological properties of the ROI may be calculated using techniques known in the art. The input to the ML model may then be a vector of its morphological properties for a particular ROI. Examples of morphological properties that may be calculated may include, but are not limited to, size, area, circularity, sharpness, elongation, aspect ratio, shape factor, perimeter, area, brightness level, void count, void area, or skeleton length. In some embodiments, input to the ML model may include both two-dimensional image data and calculated morphological properties of the ROI, and the output of the ML model may be a classification of the ROI based on the number and type of classifications (e.g., GSR or non-GSR, abrasive or soft, etc.) used in a particular application.

[0023] The classification logic 1006 may also include user-defined rules (sometimes contained in a "rules file") to apply to data generated by various analysis modes to arrive at a classification of an associated ROI that differs from the ML model that generates such classification. These additional rules may be used when further information about the ROI is available (e.g., when a portion of the physical sample corresponding to the ROI is imaged using multiple different analysis modes). For example, if the physical sample is imaged using EDS, and EDS provides information about the elemental composition of the portion of the physical sample corresponding to the ROI, the classification logic may store rules that can compare that elemental composition information (potentially in conjunction with characteristics of the ROI, such as its size, roughness, etc.) to determine the appropriate classification of the ROI. Other analysis modes may provide other types of information (e.g., crystal structure rotation information provided by EBSD) that may be part of the set of classification rules.

[0024] In some embodiments, the ML model implemented by the classification logic 1006 can be trained using previous classifications (e.g., input-output pairs including image / morphological data for individual ROIs and classifications of those ROIs). This training can begin when the module 1000 is initially deployed at a particular site for a particular application, or as soon as the application area is known and training data is available. In some embodiments, the ML model can be refined or initially trained based on data that becomes available to the module 1000 after deployment. For example, the microscope system can initially generate BSED and EDS data for each ROI (when both analysis modes are available) and generate classifications for those ROIs using user-defined rules, and then initially train an ML model using that data (BSED and BSED-derived data as inputs and classifications as outputs in a training set) to perform classifications or improve the performance of a previously trained ML model. In this way, as the classification performance of the ML model improves (as determined by the evaluation logic 1008, as discussed below), fewer ROIs may need to be analyzed using EDS for an accurate classification to be determined, resulting in significant time savings.

[0025] In some embodiments, the classification of an ROI may be “unclassified,” meaning that the proper classification of the ROI is unknown. This may occur when the ML model of the classification logic 1006 outputs an “unclassified” result (e.g., when there is low confidence in a particular classification, as determined by the evaluation logic 1008, as discussed below), when the ROI cannot be reanalyzed using the second analysis mode (e.g., when a particular microscope system is not configured to analyze using the second mode), and / or when the results of the analysis after the second analysis mode do not correspond to any known classification (e.g., according to stored rules applied by the classification logic 1006). In some such cases, when the classification of an ROI is unclassified, the classification logic 1006 may store the “unclassified” classification of the ROI and may update the classification of the ROI upon notification from the central server that a classification is available. The classification may become available under any of several circumstances, such as improving the performance of the ML model to account for previously unclassified ROI, creating a new ML model to properly classify a previously unclassified ROI, and / or creating new rules that can process previously unclassified ROI. An example of how to create a new classification for a previously unclassified ROI is described below with reference to FIG.

[0026] The evaluation logic 1008 can evaluate the classification of the ROI generated by the classification logic 1006 to determine whether the confidence in the classification is high enough for the classification to be accepted for that ROI. In some embodiments, the classification logic 1006 can generate a confidence score along with the classification using techniques known in machine learning, and the evaluation logic 1008 can compare the confidence score to a predetermined threshold to determine whether the classification of the ROI should be accepted (e.g., stored, included in a report assembled by the user interface logic 1010, etc.) or whether the ROI should be re-analyzed (e.g., using a different, more resource-intensive analysis modality such as EDS) to generate data that can be used to generate a more reliable classification of the ROI. In some embodiments, the confidence score can be a value between 0 and 1, as known in the art, although the range of confidence scores can be scaled to any desired range. The evaluation logic 1008 can compare the confidence score to a threshold (e.g., 0.95 or another value) and can accept classifications whose confidence meets or exceeds that threshold (or can reject the classifications otherwise).

[0027] In some embodiments, the evaluation logic 1008 may track the confidence of the ML model of the classification logic 1006 over time and may monitor confidence trends that indicate the performance of the ML model for a particular output classification or all classifications is decreasing or falling below a threshold. If such a performance degradation condition is identified by the evaluation logic 1008, the evaluation logic 1008 may cause the classification logic 1006 to retrain the ML model on additional data to improve performance. Having the classification logic 1006 retrain may mean having the device interface logic 1002 collect more data using both the first and second analysis modes, for which user-defined rules can be applied to generate classifications, and the resulting data and classifications can be provided to the classification logic 1006 to retrain the ML model.

[0028] The user interface logic 1010 can provide information to and / or receive information from a human user of the analysis module 1000. In some embodiments, the user interface logic 1010 can aggregate information about the physical sample into a report that includes information about ROIs identified within an image of the physical sample. The report can be provided to a user for visual display, electronic transmission, or use by a quality control or other system to help make automated or semi-automated decisions about downstream processing or handling of the physical sample or upstream parameters (e.g., whether previous processing steps were performed correctly, whether raw materials had appropriate properties, whether the quality of the produced material meets specified standards, etc.). In some embodiments in which the systems and methods disclosed herein are utilized as part of a particle analysis (PA) workflow, the report output by the user interface logic 1010 can identify all of the identifiable particles of the physical sample and their associated classifications. Some examples of reports that can be generated by the user interface logic 1010 are discussed herein.

[0029] The sample analysis module 1000 can perform any of several sample analysis methods. FIG. 4 is a flow diagram of a method 2000 for performing a sample analysis operation, according to various embodiments. In particular, method 2000 is a method for classifying microscopic components of a physical sample. While the operations of method 2000 (and other methods disclosed herein) may be illustrated with reference to particular embodiments disclosed herein (e.g., the sample analysis module 1000 discussed herein with reference to FIG. 1 , the GUI 3000 discussed herein with reference to FIG. 7 , the computing device 4000 discussed herein with reference to FIG. 8 , and / or the sample analysis system 5000 discussed herein with reference to FIG. 9 ), method 2000 (and other methods disclosed herein) may be used in any suitable setting to perform any suitable sample analysis operation. While the operations are shown in FIG. 4 once each and in a particular order, the operations may be appropriately reordered and / or repeated as needed (e.g., different operations performed may be performed in parallel, as appropriate).

[0030] At 2002, a set of ROIs within an image representing the physical sample may be generated. The image itself may be generated by the microscope system using a first analysis mode. The ROI logic 1004 of the sample analysis module 1000 may perform the operations of 2002 based on data provided by the instrument interface logic 1002 of the sample analysis module 1000. In some embodiments, generating the set of ROIs at 2002 includes applying an ML-based segmentation technique to the image. The particular ROIs identified depend on the application. For example, in some applications, the individual ROIs identified at 2002 may correspond to individual particles within the physical sample.

[0031] In 2004, an initial classification for each ROI (of the set generated in 2002) may be generated by applying the trained ML model to at least a portion of the image associated with the ROI. The classification logic 1006 of the sample analysis module 1000 may perform the operations of 2004. As described above, the set of possible classifications (e.g., two or more) that may be generated in 2004 depends on the particular application and how the ML model was trained. For example, in some law enforcement applications, the initial classification of the ROI in 2004 may be selected from a set of at least two classifications, the set of at least two classifications including GSR and not including GSR. In some automotive applications, the initial classification of the ROI in 2004 may be selected from a set of at least two classifications, the set of at least two classifications including abrasive and soft. In some embodiments, the ML model applied in 2004 may be received by the classification logic 1006 from a central server (e.g., remote computing device 5040 of FIG. 9 , described below), which may be configured to deploy the ML model to multiple microscope systems. In other embodiments, the classification logic 1006 itself can deploy the ML model to other microscope systems.

[0032] In 2006, a confidence score associated with the initial classification (generated in 2004) may be generated. The classification logic 1006 of the sample analysis module 1000 may perform the operations of 2006, and in some particular embodiments, may do so in conjunction with or simultaneously with generating the initial classification in 2004.

[0033] At 2008, it may be determined whether the initial classification of each ROI satisfies a set of one or more confidence criteria. The evaluation logic 1008 of the sample analysis module 1000 may perform the operations of 2008. In some embodiments, the operations of 2008 may include comparing the confidence score (generated in 2006) to a threshold and comparing the confidence score to the threshold. When the confidence score exceeds the threshold, the confidence criteria may be met.

[0034] If, in 2008, it is determined that the initial classification meets a confidence criterion, the method 2000 may proceed to 2010, where the initial classification may be set as the final classification for the associated ROI. The user interface logic 1010 of the sample analysis module 1000 may perform the operations of 2010.

[0035] If, in 2008, it is determined that the initial classification does not meet the confidence criterion, method 2000 may proceed to 2012, where the microscope system may be caused to re-analyze at least a portion of the sample associated with the ROI using a second analysis mode different from the first analysis mode (e.g., performing an EDS analysis after the initial BSED analysis). The instrument interface logic 1002 of the sample analysis module 1000 may perform the operations of 2012. In some embodiments, the microscope system takes less time to image the portion of the sample associated with the ROI using the first analysis mode (the mode used to generate the image analyzed in 2002) than using the second analysis mode (the mode triggered in 2012 by not meeting the confidence criterion of 2008). For example, in some embodiments, when generating a confidence score associated with the initial classification, the first analysis mode may be or may include BSED, and the second analysis mode may be or may include EDS. Because another round of imaging using the second analysis mode cannot be triggered unless the initial classification does not meet a confidence criterion, it is expected that as long as the ML model used in the initial classification in 2004 has adequate performance (e.g., after the ML model has been properly trained), not all of the ROIs generated in 2002 will need to be re-analyzed; therefore, implementation of method 2000 may require less time to analyze the physical sample than conventional methods in which all of the ROIs are imaged using the second analysis mode.

[0036] In some embodiments, the method 2000 may further include, after reanalyzing the sample in 2012, using the data generated by the reanalysis to generate a final classification of the ROI. The classification logic 1006 may include rules to apply to the data generated by the second analysis mode (potentially along with characteristics of the first image and / or ROI) to arrive at a classification of the associated ROI. For example, if the second analysis mode is EDS and reanalyzing the ROI using EDS provides information regarding the elemental composition of a portion of the physical sample corresponding to the ROI, that elemental composition information (potentially along with characteristics of the ROI, such as its size, roughness, etc.) may be compared to a set of pre-stored rules that determine an appropriate classification of the ROI.

[0037] Once method 2000 has been performed for all of the ROIs of a particular physical sample, user interface logic 1010 may report, store, electronically transmit, or otherwise use the final ROI classification for each of the ROIs. In some embodiments, user interface logic 1010 may output a classification report including the final classification of each ROI within the set of ROIs. The classification report may include any other suitable information, such as the location of each ROI, morphological characteristics (e.g., area, circularity, or roughness) of the portion of the sample corresponding to each ROI, compositional information (e.g., gunpowder, aluminum oxide, etc.) of the portion of the sample corresponding to each ROI, and / or any other suitable information.

[0038] As described above, in some embodiments of method 2000, the initial and / or final classification of the ROI may be "unclassified," meaning that the appropriate classification of the ROI is unknown (e.g., the classification is determined in 2008 to not meet the confidence criteria for an "approved" classification). In some embodiments, method 2000 may include that when the classification of the ROI is unclassified, the classification logic 100 may store the "unclassified" classification of the ROI and may update the classification of the ROI when notified by the central server that a classification is available.

[0039] As noted above, in some embodiments, the sample analysis module 1000 can generate a new ML model. Figure 5 is a flow diagram of a method 2100 for generating an ML model for classifying microscopic components of a physical sample, according to various embodiments. Although the operations are shown in Figure 5 once each and in a particular order, the operations may be appropriately reordered and / or repeated as needed (e.g., different operations performed may be performed in parallel, as appropriate).

[0040] At 2102, first data representing a sample may be received. The first data may be generated using a first analysis mode (e.g., of a microscope system). The instrument interface logic 1002 may perform the operations of 2102.

[0041] At 2104, a classification may be received. The classification may correspond to an ROI within the first data, and the classification may have been determined using the second data and a second analysis mode (e.g., of the microscope system) that is different from the first analysis mode. The classification logic 1006 may perform the operations of 2104. In some embodiments, the classification may be generated manually by a human user or may be generated using a rules-based process. In some embodiments, the microscope system takes less time to image a portion of a sample using the first analysis mode than using the second analysis mode (e.g., the first analysis mode may be or may include BSED, and the second analysis mode may be or may include EDS or another method that may generate elemental composition data for a portion of a sample). In some embodiments, individual ROIs within the first data correspond to individual particles within a physical sample. In some such embodiments, the classification received at 2104 may be selected from a set of at least two classifications (e.g., gunshot residue (GSR) and non-GSR, abrasive and soft, etc.).

[0042] At 2106, an ML model may be trained to classify ROIs in the first analysis mode data using the first data and the classification. The classification logic 1006 may perform the operations of 2106. In some embodiments, training the ML model using the first data and the classification may include training the machine learning model using the first analysis data, morphological parameters (e.g., area, circularity, roughness, etc.) generated at least in part from the first analysis data, or a combination of both. In some embodiments, the ML model of 2106 may be configured to receive as input two-dimensional images (e.g., one or more images included in the first data), one-dimensional vectors (e.g., vectors of morphological parameters), or a combination of both (e.g., two-dimensional images represented as an array and concatenated or otherwise combined with data in one-dimensional vectors of morphological parameters).

[0043] In some embodiments, the operation of 2106 may include generating a training performance score (e.g., by the classification logic 1006) after training the ML model. In such embodiments, the training performance score may be compared (e.g., by the evaluation logic 1008) to a predetermined training performance criterion. If the training performance score does not meet the training performance criterion (e.g., the classification performance of the ML model is inadequate), the ML model may be retrained (e.g., by the classification logic 1006) based on additional first data and additional corresponding classification data. When the training performance score meets the training performance criterion, the ML model may be provided (e.g., by the classification logic 1006) to generate classifications of ROIs in additional images of the physical sample, the additional images being generated by the microscope system using the first analysis mode. In this manner, the ML model need not be deployed until its classification performance is adequate.

[0044] In some embodiments, the method 2100 may include, after receiving the first data at 2102, generating an ROI within the first data. The ROI logic 1004 may generate the ROI in such embodiments. In some such embodiments, generating the ROI may include applying machine learning segmentation techniques to the first data (received at 2102).

[0045] In some embodiments, method 2100 may include, after training the ML model at 2106, deploying the ML model (e.g., to multiple microscope systems) according to any of the embodiments discussed herein (e.g., as discussed above with reference to the operation of classification logic 1006 and / or 2004 of method 2100).

[0046] As noted above, in some embodiments where the initial classification of an ROI is "unclassified," the classification logic 1006 can store the "unclassified" classification of the ROI and can update the classification of the ROI when notified by the central server that a classification is available. Figure 6 is a flow diagram of a method 2200 of creating a new classification for a previously unclassified ROI, according to various embodiments. Although the operations are shown in Figure 6 once each and in a particular order, the operations may be appropriately reordered and / or repeated as needed (e.g., different operations performed may be performed in parallel, as appropriate).

[0047] At 2202, first analysis mode data representing a set of ROIs of a sample may be received. The ROIs associated with the first analysis mode data received at 2202 may have been previously classified by a machine learning model as not corresponding to a known classification (e.g., “unclassified”), and the first analysis mode data may have been generated by a microscope system using a first analysis mode (e.g., BSED). The instrument interface logic 1002 may perform the operations of 2202. In some embodiments, the individual ROIs correspond to individual particles in a physical sample, while in other embodiments, the individual ROIs may correspond to different components or regions in the sample (e.g., specific structures or components of a biological or non-biological sample).

[0048] At 2204, the first analysis mode data may be clustered. The classification logic 1006 may perform the operations of 2204. In some embodiments, the first analysis mode data may be clustered at 2204 based on similarities between the first analysis mode data corresponding to different ROIs (e.g., the more similar the first analysis mode data associated with a first ROI is to the first analysis mode data associated with a second ROI, the more closely the two sets of first analysis mode data may be clustered). Accordingly, this disclosure may refer to clustering of ROIs associated with first analysis mode data.

[0049] At 2206, new first analysis mode data representing the new ROI may be received. The device interface logic 1002 may perform the operations of 2206.

[0050] At 2208, the new first analysis mode data may be determined to belong to a particular one of the clusters generated at 2204. This determination may be based on a comparison of the new first analysis mode data with the analysis mode data clustered at 2204, with the new first analysis mode data being included in the particular cluster with the most similar previously received first analysis mode data. The classification logic 1006 may perform the operations of 2208.

[0051] At 2210, second analysis mode data representing the new ROI may be received. The second analysis mode data may be generated by the microscope system using a second analysis mode that is different from the first analysis mode (associated with the data received in 2202). The instrument interface logic 1002 may perform the operations of 2210. In some embodiments, the microscope system may take less time to image the portion of the sample associated with the ROI using the first analysis mode than using the second analysis mode. For example, the first analysis mode may include BSED and the second analysis mode may include EDS.

[0052] At 2212, an identification of a new classification associated with a particular cluster (to which the new ROI belonged determined at 2208) may be received, and at 2214, the ROIs in the particular cluster may be reclassified to the new classification. The user interface logic 1010 may perform the operations of 2212, and the classification logic 1006 may perform the operations of 2214. In some embodiments, the identification received at 2212 may be specified by a user, while in other embodiments, the identification may be generated automatically. For example, in some embodiments, a user may review the first analysis mode data (received at 2202 and 2206) of the ROIs included in a particular cluster and, in conjunction with additional information provided by the second analysis mode data (received at 2210), determine that all previously “unclassified” ROIs in a particular cluster should be assigned a new classification (e.g., a new type of pollutant, a new type of particle, etc.), and the user may provide the new classification for the particular cluster. In some embodiments, user interface logic 1010 may facilitate operation of 2012 by providing the user with a list of ROIs and second analysis mode data within a particular cluster via a GUI (e.g., as described below with reference to FIG. 7), which may allow the user to type or otherwise input new classifications that may be assigned to all or some of the ROIs within a particular cluster. In other embodiments, new classifications may be identified automatically by module 1000 based on other data available to module 1000.

[0053] In some embodiments, method 2200 may further include outputting a classification report including the classification of the individual ROIs of the physical sample after reclassifying the ROIs at 2212. The user interface logic 1010 may perform such operations. The classification report may include the locations of the individual ROIs, morphological characteristics of the portions of the physical sample corresponding to the individual ROIs, and / or any other available or derived information about the physical sample.

[0054] In some embodiments, the methods disclosed herein may be performed by individual microscope systems and / or by a central server in communication with multiple microscope systems. For example, in some implementations, the central server may receive classification-related data and train and retrain ML models to perform classifications; once the ML model's performance exceeds a threshold, the ML model may be pushed to different microscope systems in communication with the server. The different microscope systems may then use the ML model to perform classifications of components of the sample being analyzed. In some embodiments, method 2200 of identifying an appropriate classification for a previously unclassified ROI may be performed by a central server, which may be configured to update the ML model and / or previous classifications with new classifications once available.

[0055] As discussed herein with reference to the user interface logic 1010 of the sample analysis module 1000, the sample analysis methods disclosed herein may include interactions with a human user (e.g., via the user local computing device 5020 discussed herein with reference to FIG. 9). These interactions may include providing information to the user (e.g., information about the sample being analyzed, or information about other tests or measurements performed by a scientific instrument such as the scientific instrument 5010 of FIG. 9 described below, information retrieved from a local or remote database, or other information), or providing options for the user to enter commands (e.g., to control the operation of a scientific instrument such as the scientific instrument 5010 of FIG. 9 or to control the analysis of data generated by the scientific instrument), queries (e.g., to a local or remote database), or other information. In some embodiments, these interactions may be performed through a GUI that includes a visual display on a display device (e.g., display device 4010 discussed herein with reference to FIG. 8) that provides output to the user and / or prompts the user to provide input (e.g., via one or more input devices such as a keyboard, mouse, trackpad, or touchscreen included in other I / O devices 4012 discussed herein with reference to FIG. 8). The sample analysis systems disclosed herein may include any suitable GUI for interaction with a user.

[0056] 7 depicts an exemplary GUI 3000 that may be used in implementing some or all of the sample analysis methods disclosed herein, according to various embodiments. As mentioned above, the GUI 3000 may be provided on a display device (e.g., display device 4010 discussed herein with reference to FIG. 8) of a computing device (e.g., computing device 4000 discussed herein with reference to FIG. 8) of a sample analysis system (e.g., sample analysis system 5000 discussed herein with reference to FIG. 9), and a user may interact with the GUI 3000 using any suitable input device (e.g., any of the input devices included in other I / O devices 4012 discussed herein with reference to FIG. 8) and input technique (e.g., cursor movement, motion capture, facial recognition, gesture detection, voice recognition, button actuation, etc.).

[0057] GUI 3000 may include a data display region 3002, a data analysis region 3004, a scientific instrument control region 3006, and a settings region 3008. The particular number and arrangement of regions shown in Figure 7 are merely exemplary, and any number and arrangement of regions containing any desired features may be included in GUI 3000.

[0058] The data display area 3002 may display data generated by a scientific instrument (e.g., the scientific instrument 5010 discussed herein with reference to FIG. 9). For example, the data display area 3002 may display images or other data (e.g., EDS data) representative of a physical sample (e.g., data generated by one or more microscope systems and received by the instrument interface logic 1002 of the sample analysis module 1000). In some embodiments, one or more ROIs identified by the ROI logic 1004 may be displayed on or along with associated image data in the data display area 3002.

[0059] The data analysis area 3004 can display the results of the data analysis (e.g., the results of analyzing the data illustrated in the data display area 3002 and / or other data). For example, the data analysis area 3004 can display calculated morphological properties of individual ROIs, the location of individual ROIs, and the classification of the ROIs (e.g., generated by the classification logic 1006 and evaluated by the evaluation logic 1008). In some embodiments, the data analysis area 3004 can include a classification report, as discussed herein, or an option for the user to download or send the classification report. In some embodiments, the data display area 3002 and the data analysis area 3004 can be combined in the GUI 3000 (e.g., to include data output from a scientific instrument and several analyses of the data in a common graph or area).

[0060] The scientific instrument control area 3006 may include options that allow a user to control a scientific instrument (e.g., the scientific instrument 5010 described herein with reference to FIG. 9). For example, the scientific instrument control area 3006 may include controls for the instrument interface logic 1002 (e.g., for initiating a second analysis mode for analyzing a portion of a sample, as discussed herein).

[0061] Settings area 3008 may include options that enable a user to control features and functionality of GUI 3000 (and / or other GUIs) and / or perform common computing operations (e.g., saving data on a storage device such as storage device 4004 discussed herein with reference to FIG. 8 , sending data to another user, labeling data, etc.) with respect to data display area 3002 and data analysis area 3004. For example, settings area 3008 may include controls for updating an ML model, controls for deploying an ML model from a server, or other types of controls.

[0062] As noted above, the sample analysis module 1000 may be implemented by one or more computing devices. Figure 8 is a block diagram of a computing device 4000 that may perform some or all of the sample analysis methods disclosed herein, according to various embodiments. In some embodiments, the sample analysis module 1000 may be implemented by a single computing device 4000 or by multiple computing devices 4000. Furthermore, as described below, the computing device 4000 (or multiple computing devices 4000) that implements the sample analysis module 1000 may be part of one or more of the scientific instrument 5010, user local computing device 5020, service local computing device 5030, or remote computing device 5040 of Figure 9.

[0063] 8 is shown as having several components, any one or more of which may be omitted or duplicated as appropriate for the application and configuration. In some embodiments, some or all of the components included in computing device 4000 may be mounted on one or more motherboards and enclosed in a housing (e.g., comprising plastic, metal, and / or other materials). In some embodiments, several of these components may be fabricated on a single system-on-a-chip (SoC) (e.g., an SoC may include one or more processing devices 4002 and one or more storage devices 4004). 8, but may include interface circuitry (not shown) for coupling to one or more components using any suitable interface (e.g., a Universal Serial Bus (USB) interface, a High-Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface). For example, computing device 4000 may not include display device 4010, but may include display device interface circuitry (e.g., connectors and driver circuitry) to which display device 4010 may be coupled.

[0064] The computing device 4000 may include a processing device 4002 (e.g., one or more processing devices). As used herein, the term "processing device" may refer to any device or portion of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that may be stored in registers and / or memory. The processing device 4002 may include one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), cryptographic processors (dedicated processors that execute cryptographic algorithms in hardware), server processors, or any other suitable processing devices.

[0065] The computing device 4000 may include a storage device 4004 (e.g., one or more storage devices). The storage device 4004 may include one or more memory devices, such as random access memory (RAM) (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive-bridging RAM (CBRAM) devices), hard drive-type memory devices, solid-state memory devices, networked drives, cloud drives, or any combination of memory devices. In some embodiments, the storage device 4004 may include memory that shares a die with the processing device 4002. In such embodiments, the memory may be used as cache memory and may include, for example, embedded dynamic random access memory (eDRAM) or spin transfer torque magnetic random access memory (STT-MRAM). In some embodiments, storage device 4004 may include a non-transitory computer-readable medium having instructions that, when executed by one or more processing devices (e.g., processing device 4002), cause computing device 4000 to perform any suitable method or portion thereof of the methods disclosed herein.

[0066] The computing device 4000 may include an interface device 4006 (e.g., one or more interface devices 4006). The interface device 4006 may include one or more communication chips, connectors, and / or other hardware and software for managing communications between the computing device 4000 and other computing devices. For example, the interface device 4006 may include circuitry for managing wireless communications for data transfer to and from the computing device 4000. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communication channels, etc. that may communicate data through the use of modulated electromagnetic radiation over a non-solid medium. This term does not imply that the associated devices do not include any wiring, although in some embodiments they may not. The circuitry included in interface device 4006 for managing wireless communications may implement any of several wireless standards or protocols, including, but not limited to, Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Long-Term Evolution (LTE) projects with any amendments, updates, and / or revisions (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also referred to as "3GPP®2"), etc.). In some embodiments, the circuitry included in interface device 4006 for managing wireless communications may implement any of several wireless standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Long-Term Evolution (LTE) projects with any amendments, updates, and / or revisions (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also referred to as "3GPP®2"), etc.). The interface device 4006 may operate in accordance with a Global Positioning System (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. In some embodiments, the circuitry included in the interface device 4006 for managing wireless communications may operate in accordance with an Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, the circuitry included in interface device 4006 for managing wireless communications may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols designated as 3G, 4G, 5G, and beyond. In some embodiments, interface device 4006 may include one or more antennas (e.g., one or more antenna arrays) for receiving and / or transmitting wireless communications.

[0067] In some embodiments, the interface device 4006 may include circuitry for managing wired communications, such as electrical, optical, or any other suitable communications protocol. For example, the interface device 4006 may include circuitry to support communications according to Ethernet technology. In some embodiments, the interface device 4006 may support both wireless and wired communications and / or multiple wired and / or wireless communications protocols. For example, a first set of circuits in the interface device 4006 may be dedicated to short-range wireless communications, such as Wi-Fi or Bluetooth, and a second set of circuits in the interface device 4006 may be dedicated to long-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first set of circuits in the interface device 4006 may be dedicated to wireless communications and the second set of circuits in the interface device 4006 may be dedicated to wired communications.

[0068] Computing device 4000 may include battery / power circuitry 4008. Battery / power circuitry 4008 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of computing device 4000 to an energy source separate from computing device 4000 (e.g., AC line power).

[0069] The computing device 4000 may include a display device 4010 (e.g., multiple display devices). The display device 4010 may include any visual indicator, such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0070] The computing device 4000 may include other input / output (I / O) devices 4012. The other I / O devices 4012 may include, for example, one or more audio output devices (e.g., speakers, headsets, earphones, alarms, etc.), one or more audio input devices (e.g., microphones or microphone arrays), a location device (e.g., a GPS device that communicates with a satellite-based system to receive the location of the computing device 4000, as is known in the art), an audio codec, a video codec, a printer, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, accelerometers, gyroscopes, etc.), an image capture device such as a camera, a keyboard, a cursor control device such as a mouse, stylus, trackball, or touchpad, a barcode reader, a Quick Response (QR) code reader, or a radio frequency identification (RFID) reader.

[0071] The computing device 4000 may have any form factor suitable for its application and configuration, such as a handheld or mobile computing device (e.g., a mobile phone, smartphone, mobile internet device, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), ultra-mobile personal computer, etc.), a desktop computing device, or a server computing device or other networked computing component.

[0072] One or more computing devices implementing any of the sample analysis modules or methods disclosed herein may be part of a sample analysis system. Figure 9 is a block diagram of an exemplary sample analysis system 5000 in which some or all of the sample analysis methods disclosed herein may be implemented, according to various embodiments. The sample analysis modules and methods disclosed herein (e.g., sample analysis module 1000 of Figure 1 and method 2000 of Figure 4) may be implemented by one or more of the scientific instrument 5010, user local computing device 5020, service local computing device 5030, or remote computing device 5040 of the sample analysis system 5000.

[0073] Any of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may include any of the embodiments of the computing device 4000 discussed herein with reference to FIG. 8, and any of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may take the form of any suitable embodiment of the embodiments of the computing device 4000 discussed herein with reference to FIG. 8.

[0074] The scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may each include a processing device 5002, a storage device 5004, and an interface device 5006. The processing device 5002 may take any suitable form, including any form of the processing device 4002 described herein with reference to Figure 8, and the processing devices 5002 included in different ones of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may take the same form or different forms. The storage device 5004 may take any suitable form, including any form of the storage device 4004 discussed herein with reference to Figure 8, and the storage devices 5004 included in different ones of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may take the same form or different forms. The interface device 5006 may take any suitable form, including any of the forms of the interface device 4006 discussed herein with reference to FIG. 8, and the interface devices 5006 included in different ones of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, or the remote computing device 5040 may take the same or different forms.

[0075] The scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, and the remote computing device 5040 may communicate with other elements of the sample analysis system 5000 via communication paths 5008. The communication paths 5008 may communicatively couple the interface devices 5006 of different ones of the elements of the sample analysis system 5000, as shown, and may be wired or wireless communication paths (e.g., via any of the communication techniques discussed herein with reference to the interface device 4006 of the computing device 4000 of FIG. 8). While the particular sample analysis system 5000 shown in FIG. 9 includes communication paths between each pair of the scientific instrument 5010, the user local computing device 5020, the service local computing device 5030, and the remote computing device 5040, this "fully coupled" implementation is merely exemplary, and in various embodiments, various ones of the communication paths 5008 may not be present. For example, in some embodiments, the service local computing device 5030 may not have a direct communication path 5008 between its interface device 5006 and the interface device 5006 of the scientific instrument 5010, but instead may communicate with the scientific instrument 5010 via a communication path 5008 between the service local computing device 5030 and the user local computing device 5020, and a communication path 5008 between the user local computing device 5020 and the scientific instrument 5010.

[0076] Scientific instrument 5010 may include any suitable scientific instrument, such as a charged particle microscope (eg, an electron microscope), an optical microscope, a spectroscopic device, or any other suitable analytical instrument.

[0077] The user local computing device 5020 may be a computing device that is local to a user of the scientific instrument 5010 (e.g., according to any of the embodiments of the computing device 4000 discussed herein). In some embodiments, the user local computing device 5020 may be, but need not be, local to the scientific instrument 5010. For example, a user local computing device 5020 in a user's home or office may be remote from, but able to communicate with, the scientific instrument 5010 such that the user may use the user local computing device 5020 to control and / or access data from the scientific instrument 5010. In some embodiments, the user local computing device 5020 may be a laptop, smartphone, or tablet device. In some embodiments, the user local computing device 5020 may be a portable computing device.

[0078] The servicing local computing device 5030 may be a computing device (e.g., according to any of the embodiments of computing device 4000 discussed herein) that is local to an entity that provides services to the scientific instrument 5010. For example, the servicing local computing device 5030 may be local to the manufacturer of the scientific instrument 5010 or a third-party service company. In some embodiments, the servicing local computing device 5030 may communicate with the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., via a direct communication path 5008 or via multiple "indirect" communication paths 5008, as described above) to receive data regarding the operation of the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., results of self-tests of the scientific instrument 5010, calibration coefficients used by the scientific instrument 5010, measurements of sensors associated with the scientific instrument 5010, etc.). In some embodiments, the service local computing device 5030 can communicate with the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., via a direct communication path 5008 or via multiple "indirect" communication paths 5008, as described above) and transmit data to the scientific instrument 5010, the user local computing device 5020, and / or the remote computing device 5040 (e.g., to update programmed instructions such as firmware in the scientific instrument 5010, to initiate the performance of a test or calibration sequence in the scientific instrument 5010, to update programmed instructions such as software in the user local computing device 5020 or the remote computing device 5040, etc.).A user of the scientific instrument 5010 can use the scientific instrument 5010 or the user local computing device 5020 to communicate with the service local computing device 5030 to report problems with the scientific instrument 5010 or the user local computing device 5020, request a technician visit to improve the operation of the scientific instrument 5010, order consumables or replacement parts related to the scientific instrument 5010, or for other purposes.

[0079] The remote computing device 5040 may be a computing device (e.g., according to any of the embodiments of the computing device 4000 discussed herein) that is remote from the scientific instrument 5010 and / or the user local computing device 5020. In some embodiments, the remote computing device 5040 may be included in a data center or other large-scale server environment. In some embodiments, the remote computing device 5040 may include network-attached storage (e.g., as part of the storage device 5004). The remote computing device 5040 may store data generated by the scientific instrument 5010, perform analysis of the data generated by the scientific instrument 5010 (e.g., according to programmed instructions), facilitate communication between the user local computing device 5020 and the scientific instrument 5010, and / or facilitate communication between the service local computing device 5030 and the scientific instrument 5010. In some embodiments, the remote computing device may be a server that manages and deploys ML models and implements new classifications of previously unclassified ROIs, as described herein.

[0080] In some embodiments, one or more of the elements of the sample analysis system 5000 illustrated in Figure 9 may not be present. Furthermore, in some embodiments, more than one of various of the elements of the sample analysis system 5000 of Figure 9 may be present. For example, the sample analysis system 5000 may include multiple user local computing devices 5020 (e.g., different user local computing devices 5020 associated with different users or in different locations). In another example, the sample analysis system 5000 may include multiple scientific instruments 5010 that all communicate with a servicing local computing device 5030 and / or a remote computing device 5040. In such an embodiment, the servicing local computing device 5030 may monitor these multiple scientific instruments 5010, and the servicing local computing device 5030 may trigger updates or "broadcast" other information to the multiple scientific instruments 5010 simultaneously. Different ones of the scientific instruments 5010 in the sample analysis system 5000 can be located in close proximity to each other (e.g., in the same room) or farther away from each other (e.g., on different floors of a building, in different buildings, in different cities, etc.). In some embodiments, the scientific instruments 5010 may be connected to an Internet-of-Things (IoT) stack that enables command and control of the scientific instruments 5010 through web-based applications, virtual or augmented reality applications, mobile applications, and / or desktop applications. Any of these applications may be accessed by a user operating a user local computing device 5020 that communicates with the scientific instruments 5010 by way of an intervening remote computing device 5040. In some embodiments, the scientific instruments 5010 may be sold by a manufacturer as part of a local scientific instrument computing unit 5012, along with one or more associated user local computing devices 5020.

[0081] The following paragraphs provide various examples of the embodiments disclosed herein.

[0082] Example A includes any of the sample analysis modules disclosed herein.

[0083] Example B includes any of the methods disclosed herein.

[0084] Example C includes any of the GUIs disclosed herein.

[0085] Example D includes any of the sample analysis computing devices and systems disclosed herein.

[0086] Example 1 is a method for classifying microscopic components of a physical sample, including generating a set of regions of interest (ROIs) in an image representing the physical sample, the image being generated by a microscope system using a first analysis mode; generating an initial classification of the ROIs by applying a trained machine learning model to at least a portion of the image associated with the ROIs; generating a confidence score associated with the initial classification; and causing the microscope system to re-analyze at least the portion of the sample associated with the ROIs using a second analysis mode different from the first analysis mode when the confidence score for the initial classification of the ROIs does not satisfy a set of confidence criteria.

[0087] Example 2 includes the subject matter of Example 1 and further specifies that the microscope system takes less time to image the portion of the sample associated with the ROI using the first analysis mode than using the second analysis mode.

[0088] Example 3 includes the subject matter of any of Examples 1-2, and further specifies that the first analysis mode includes backscattered electron detection (BSED).

[0089] Example 4 includes the subject matter of Example 3, and further specifies that the second analysis mode includes energy dispersive spectroscopy (EDS).

[0090] Example 5 includes the subject matter of any of Examples 1-4, further specifying that some but not all of the ROI are reanalyzed using the second analysis mode.

[0091] Example 6 includes the subject matter of any of Examples 1-5, further specifying that generating the set of ROIs includes applying a machine learning segmentation technique to the image.

[0092] Example 7 includes the subject matter of any of Examples 1-6, further specifying that individual ROIs in the image correspond to individual particles in the physical sample.

[0093] Example 8 includes the subject matter of Example 7, further specifying that the initial classification of the ROI is selected from a set of at least two classifications, the set of at least two classifications including gunshot residue (GSR) and not including GSR.

[0094] Example 9 includes the subject matter of example 7, further specifying that the initial classification of the ROI is selected from a set of at least two classifications, the set of at least two classifications comprising abrasive and soft.

[0095] Example 10 includes the subject matter of any of Examples 1-9, further including using data generated by reanalysis to generate a final classification of the ROI when the confidence score for the initial classification of the ROI does not meet the set of confidence criteria.

[0096] Example 11 includes the subject matter of Example 10, and further specifies that the data generated by the reanalysis includes elemental composition data.

[0097] Example 12 includes the subject matter of any of Examples 10-11, further including using the initial classification of the ROI as the final classification of the ROI when the confidence score for the initial classification of the ROI meets a set of confidence criteria.

[0098] Example 13 includes the subject matter of example 12, further including outputting a classification report including the final classification of each ROI in the set of ROIs.

[0099] Example 14 includes the subject matter of Example 13, and further specifies that the classification report includes the location of each ROI.

[0100] Example 15 includes the subject matter of any of Examples 13-14, further specifying that the classification report includes morphological characteristics of the portions of the sample corresponding to each ROI.

[0101] Example 16 includes the subject matter of example 15, further specifying that the morphological characteristic includes area, circularity, or roughness.

[0102] Example 17 includes the subject matter of any of Examples 1-16, and further includes receiving a machine learning model from a central server before generating the initial classification of the ROI by applying the trained machine learning model.

[0103] Example 18 includes the subject matter of example 17, further specifying that the central server deploys the machine learning model to multiple microscope systems.

[0104] Example 19 includes the subject matter of any of Examples 1-18, further specifying that the initial classification of the ROI is selected from a set of at least two classifications, and the set of at least two classifications includes unclassified.

[0105] Example 20 includes the subject matter of example 19, and further includes receiving a final classification of the ROI when notified by the central server that a classification is available when the initial classification of the ROI is unclassified.

[0106] Example 21 is a method for generating a machine learning model for classifying microscopic components of a physical sample, comprising: receiving first data representing the physical sample, the first data generated by a microscope system using a first analysis mode; receiving classification data representing a classification of a region of interest (ROI) in the first data, the classification data generated based at least in part on second data of the physical sample, different from the first data, the second data generated by the microscope system using a second analysis mode different from the first analysis mode; and training a machine learning model using the data representing the first analysis data and the classification data, but without using the second data, to generate a classification of the ROI in the data generated by the microscope system using the first analysis mode.

[0107] Example 22 includes the subject matter of example 21, further including, after training the machine learning model, generating a training performance score.

[0108] Example 23 includes the subject matter of example 22, further including retraining the machine learning model based on the additional first data and the additional corresponding classification data when the training performance score does not meet the training performance criterion.

[0109] Example 24 includes the subject matter of example 22, and further includes, when the training performance score reaches the training performance criterion, providing the machine learning model to generate a classification of the ROI in an additional image of the physical sample, the additional image generated by the microscope system using the first analysis mode.

[0110] Example 25 includes the subject matter of any of Examples 21-24, further specifying that the microscope system takes less time to image the portion of the sample using the first analysis mode than using the second analysis mode.

[0111] Example 26 includes the subject matter of any of Examples 21-25, further specifying that the first analysis mode includes backscattered electron detection (BSED).

[0112] Example 27 includes the subject matter of Example 26, and further specifies that the second analysis mode includes energy dispersive spectroscopy (EDS).

[0113] Example 28 includes the subject matter of any of Examples 21-27, further including generating an ROI in the first data.

[0114] Example 29 includes the subject matter of Example 28, and further specifies that generating the ROI includes applying a machine learning segmentation technique to the first data.

[0115] Example 30 includes the subject matter of any of Examples 21-29, further specifying that the individual ROIs in the first data correspond to individual particles in the physical sample.

[0116] Example 31 includes the subject matter of any of Examples 21-30, further specifying that the classifications generated by the machine learning model are selected from a set of at least two classifications, and the set of at least two classifications includes gunshot residue (GSR) and does not include GSR.

[0117] Example 32 includes the subject matter of any of Examples 21-30, further specifying that the classification generated by the machine learning model is selected from a set of at least two classifications, and the set of at least two classifications includes abrasive and soft.

[0118] Example 33 includes the subject matter of any of Examples 21-32, and further specifies that the second data includes elemental composition data.

[0119] Example 34 includes the subject matter of any of Examples 21-33, further specifying that training the machine learning model using data representative of the first analytical data includes training the machine learning model using the first analytical data.

[0120] Example 35 includes the subject matter of any of Examples 21-33, further specifying that training the machine learning model using data representative of the first analytical data includes training the machine learning model using morphological parameters generated at least in part from the first analytical data.

[0121] Example 36 includes the subject matter of example 35, further specifying that the morphological parameter includes area, circularity, or roughness.

[0122] Example 37 includes the subject matter of any of Examples 21-36, further specifying that the machine learning model is configured to receive a two-dimensional image as input.

[0123] Example 38 includes the subject matter of any of Examples 21-36, further specifying that the machine learning model is configured to receive a one-dimensional vector as input.

[0124] Example 39 includes the subject matter of example 38, and further specifies that the one-dimensional vector includes at least some morphological parameters.

[0125] Example 40 includes the subject matter of any of examples 21-39, further including deploying the machine learning model to a plurality of microscope systems.

[0126] Example 41 is a method for classifying microscopic components of a physical sample, comprising: receiving first analysis mode data representing a set of regions of interest (ROIs) in an image representing the physical sample, the ROIs having been previously classified by a machine learning model as not corresponding to a known classification, the first analysis mode data having been generated by a microscope system using a first analysis mode; clustering the first analysis mode data based on similarities between the first analysis mode data corresponding to the different ROIs; receiving new first analysis mode data representing the new ROIs; determining that the new first analysis mode data indicates that the new ROIs belong to a particular cluster; receiving second analysis mode data representing the new ROIs, the second analysis mode data having been generated by the microscope system using a second analysis mode different from the first analysis mode; providing a user with a list of the ROIs in the particular cluster and the second analysis mode data; receiving an identification of a new classification associated with the particular cluster; and reclassifying the ROIs in the particular cluster to the new classification.

[0127] Example 42 includes the subject matter of Example 41, and further specifies that the microscope system requires less time to image the portion of the sample associated with the ROI using the first analysis mode than using the second analysis mode.

[0128] Example 43 includes the subject matter of any of Examples 41-42, further specifying that the first analysis mode includes backscattered electron detection (BSED).

[0129] Example 44 includes the subject matter of Example 43, and further specifies that the second analysis mode includes energy dispersive spectroscopy (EDS).

[0130] Example 45 includes the subject matter of any of Examples 41-44, further specifying that each ROI corresponds to an individual particle in the physical sample.

[0131] Example 46 includes the subject matter of any of Examples 41-45, further including, after reclassifying the ROIs, outputting a classification report including the classification of each ROI of the physical sample.

[0132] Example 47 includes the subject matter of Example 46, and further specifies that the classification report includes the location of each ROI.

[0133] Example 48 includes the subject matter of any of Examples 46-47, further specifying that the classification report includes morphological characteristics of the portions of the physical sample corresponding to each ROI.

[0134] Example 49 includes the subject matter of any of Examples 41 to 48, and further specifies that the microscope system is a charged particle microscope system.

[0135] Example 50 includes the subject matter of any of Examples 41-49, further specifying that the physical sample is a non-biological sample.

[0136] Example 51 includes the subject matter of any of Examples 41-49, and further specifies that receiving identification information of the new classification includes receiving a user designation of the new classification.

[0137] Example 52 includes the subject matter of example 51, and further specifies that the user specification is received via a graphical user interface.

Claims

1. 1. A method for classifying microscopic components of a physical sample, comprising: generating a set of regions of interest (ROIs) in images representative of the physical sample, the images being generated by a microscope system using a first analysis mode; generating an initial classification of the ROI by applying a trained machine learning model to at least a portion of the image associated with the ROI; generating a confidence score associated with the initial classification; and when the confidence score for the initial classification of an ROI does not meet a set of confidence criteria, causing the microscope system to re-analyze at least a portion of the physical sample associated with the ROI using a second analysis mode different from the first analysis mode. method.

2. 2. The method of claim 1, wherein the microscope system takes less time to image the portion of the physical sample associated with the ROI using the first analysis mode than using the second analysis mode.

3. The method of claim 1 , wherein the first analysis mode comprises backscattered electron detection (BSED).

4. The method of claim 3 , wherein the second analysis mode comprises energy dispersive spectroscopy (EDS).

5. The method of claim 1 , wherein some but not all of the ROI is reanalyzed using the second analysis mode.

6. The method of claim 1 , wherein individual ROIs in the image correspond to individual particles in the physical sample.

7. 2. The method of claim 1, further comprising: when the confidence score for the initial classification of an ROI does not meet the set of confidence criteria, using data generated by the reanalysis to generate a final classification of the ROI.

8. The method of claim 7 , further comprising: when the confidence score for the initial classification of an ROI meets a set of confidence criteria, using the initial classification of the ROI as the final classification of the ROI.

9. The method of claim 8 , further comprising outputting a classification report including the final classification of each ROI in the set of ROIs.

10. 1. A method for generating a machine learning model for classifying microscopic components of a physical sample, comprising: receiving first data representative of the physical sample, the first data being generated by a microscope system using a first analysis mode; receiving classification data representing a classification of a region of interest (ROI) in the first data, the classification data generated based at least in part on second data of the physical sample, different from the first data, the second data generated by the microscope system using a second analysis mode different from the first analysis mode; training a machine learning model using data representative of the first data and the classification data, but without using the second data, to generate a classification of ROIs in data generated by the microscope system using the first analysis mode. method.

11. The method of claim 10 , further comprising generating a training performance score after training the machine learning model.

12. 12. The method of claim 11, further comprising retraining the machine learning model based on additional first data and additional corresponding classification data when the training performance score does not meet a training performance criterion.

13. 12. The method of claim 11, further comprising: when the training performance score reaches a training performance criterion, providing the machine learning model to generate a classification of an ROI in an additional image of a physical sample, the additional image being generated by the microscope system using the first analysis mode.

14. 11. The method of claim 10, wherein training the machine learning model using data representative of the first data comprises training the machine learning model using the first data.

15. 11. The method of claim 10, wherein training the machine learning model using data representative of the first data comprises training the machine learning model using morphological parameters generated at least in part from the first data.

16. The method of claim 10 , further comprising deploying the machine learning model to multiple microscope systems.

17. 1. A method for classifying microscopic components of a physical sample, comprising: receiving first analysis mode data representing a set of regions of interest (ROIs) in an image representing a physical sample, the ROIs having been previously classified by a machine learning model as not corresponding to a known classification, the first analysis mode data having been generated by a microscope system using a first analysis mode; clustering the first analysis mode data based on similarities between the first analysis mode data corresponding to different ROIs; receiving new first analysis mode data representing a new ROI; determining that the new first analysis mode data indicates that the new ROI belongs to a particular cluster; receiving second analysis mode data representative of the new ROI, the second analysis mode data generated by a microscope system using a second analysis mode different from the first analysis mode; providing a user with a list of ROIs within the particular cluster and the second analysis mode data; receiving an identification of a new classification associated with the particular cluster; reclassifying the ROIs within the particular cluster into the new classification. method.

18. 20. The method of claim 17, wherein the physical sample is a non-biological sample.

19. The method of claim 17 , wherein receiving the identification of the new classification comprises receiving a user designation of the new classification.

20. The method of claim 19 , wherein the user specification is received via a graphical user interface.

Images