Large language model assistance for charged-particle microscope operation

Abstract

To provide systems / techniques for facilitating large language model assistance for charged-particle microscope operation.SOLUTION: In various embodiments, a system can access a natural language instruction associated with a charged-particle microscope, where the natural language instruction can request that the charged-particle microscope undergoes a configurable settings adjustment or performs an automated task. In various aspects, the system can cause, in response to the natural language instruction, the charged-particle microscope to capture, according to a default microscopy protocol, an image or an energy spectrum of a specimen that is currently loaded on a stage of the charged-particle microscope. In various instances, the system can execute a large language model on both the natural language instruction and the image or energy spectrum of the specimen, thereby yielding a natural language response that indicates how implementing the natural language instruction would affect the specimen.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The technical field of charged particle microscopes has historically been limited by the complexity of their operation, which can prevent users from interacting with them efficiently or intuitively. Summary of the Invention

[0002] The following presents a summary to provide a basic understanding of one or more embodiments. This summary is not intended to identify key or critical elements or to delineate the scope of any particular embodiments or the scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to the more detailed description that is presented later. In one or more embodiments described herein, a device, system, computer-implemented method, apparatus, or computer program product that facilitates large-scale language model assistance for charged particle microscope operations is described.

[0003] According to one or more embodiments, a system is provided. The system may include a non-transitory computer-readable memory capable of storing computer-executable components. The system may further include a processor operably coupled to the non-transitory computer-readable memory and capable of executing the computer-executable components stored in the non-transitory computer-readable memory. In various embodiments, the computer-executable component may include an access component capable of accessing natural language instructions provided by a user of the charged particle microscope, the natural language instructions may request or instruct the charged particle microscope to undergo configurable setting adjustments or to perform an automated task. In various aspects, the computer-executable component may include a state component capable of, in response to receiving the natural language instructions, causing the charged particle microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol. In various instances, the computer-executable component may include a model component capable of running a large-scale language model on both the natural language instructions and the image or energy spectrum of the sample, thereby generating a natural language response capable of indicating how implementing the natural language instructions will affect the sample.

[0004] According to one or more embodiments, a computer-implemented method is provided. In various embodiments, the computer-implemented method can include accessing, by a device operatively coupled to a processor, natural language instructions provided by a user of the charged particle microscope, the natural language instructions may request or instruct the charged particle microscope to undergo a configurable setting adjustment or to perform an automated task. In various aspects, the computer-implemented method can include, in response to receiving the natural language instructions, causing the device to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol. In various cases, the computer-implemented method can include the device running a large-scale language model on both the natural language instructions and the image or energy spectrum of the sample, thereby generating a natural language response that can indicate how implementing the natural language instructions will affect the sample.

[0005] According to one or more embodiments, a computer program product is provided for facilitating large-scale language model assistance for charged particle microscope operations. In various embodiments, the computer program product can include a non-transitory computer-readable memory having program instructions embodied therein. In various aspects, the program instructions can be executable by the processor to cause the processor to access plain text commands provided by a user of the scanning electron microscope, the plain text commands may request the scanning electron microscope to perform a specified microscope operation. In various instances, the program instructions can be executable by the processor to, in response to receiving the plain text commands, cause the scanning electron microscope to capture an image or energy spectrum of a sample currently loaded on the stage of the scanning electron microscope via a default microscopy protocol. In various cases, the program instructions may be executable by the processor to cause the processor to execute a large-scale language model against both the plain-text command and the image or energy spectrum of the sample, the large-scale language model being capable of generating as an output a plain-text response indicating whether a specified microscopy operation will damage the sample. In various aspects, the program instructions may be executable by the processor to cause the processor to visually or audibly render the plain-text response on an electronic display or electronic speaker associated with the scanning electron microscope. [Brief explanation of the drawings]

[0006] Various embodiments will be readily understood from the following detailed description taken in conjunction with the accompanying drawings. To facilitate this description, like reference numerals refer to like structural elements. The embodiments are illustrated in the drawings by way of example, and not by way of limitation. The drawings are not necessarily drawn to scale. [Figure 1]1 illustrates an exemplary, non-limiting block diagram of a scientific instrument module according to various embodiments described herein. [Figure 2] 1 illustrates an exemplary, non-limiting flow diagram of a computer-implemented method according to various embodiments described herein. [Figure 3] 1 illustrates a block diagram of an exemplary, non-limiting system that facilitates large-scale language model assistance for charged particle microscopy operations, in accordance with one or more embodiments described herein. [Figure 4] FIG. 1 illustrates a block diagram of an exemplary, non-limiting system including a sample image, a sample energy spectrum, and a current microscope health state that facilitates large-scale language model assistance for charged particle microscope operation, according to one or more embodiments described herein. [Figure 5] FIG. 1 illustrates an exemplary, non-limiting block diagram showing how a sample image or a sample energy spectrum may be obtained according to one or more embodiments described herein. [Figure 6] 1 illustrates an exemplary, non-limiting block diagram showing how the current microscope health state may be obtained, according to one or more embodiments described herein. [Figure 7] 1 illustrates a block diagram of an exemplary, non-limiting system including a set of relevant documents, a set of inference task results, and a set of simulation results that facilitate large-scale language model assistance for charged particle microscopy operations, according to one or more embodiments described herein. [Figure 8] 1 illustrates an exemplary, non-limiting block diagram showing how a set of relevant documents, a set of inference task results, and a set of simulation results may be obtained, according to one or more embodiments described herein. [Figure 9] 1 illustrates an exemplary, non-limiting block diagram showing how a set of relevant documents, a set of inference task results, and a set of simulation results may be obtained, according to one or more embodiments described herein. [Figure 10]1 illustrates an exemplary, non-limiting block diagram showing how a set of relevant documents, a set of inference task results, and a set of simulation results may be obtained, according to one or more embodiments described herein. [Figure 11] 1 illustrates an exemplary, non-limiting block diagram showing how a set of relevant documents, a set of inference task results, and a set of simulation results may be obtained, according to one or more embodiments described herein. [Figure 12] 1 illustrates an exemplary, non-limiting block diagram showing how a set of relevant documents, a set of inference task results, and a set of simulation results may be obtained, according to one or more embodiments described herein. [Figure 13] 1 illustrates a block diagram of an exemplary, non-limiting system including natural language responses and synthesized code that facilitates large-scale language model assistance for charged particle microscope operations, according to one or more embodiments described herein. [Figure 14] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 15] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 16] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 17] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 18] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 19] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 20] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 21] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 22] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 23] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 24] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 25] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 26] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 27] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 28]1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 29] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 30] 1 illustrates an exemplary, non-limiting block diagram showing how a natural language response and synthesized code can be obtained according to one or more embodiments described herein. [Figure 31] 1 illustrates an exemplary, non-limiting block diagram showing how various artificial intelligence models can be trained in accordance with one or more embodiments described herein. [Figure 32] 1 illustrates an exemplary, non-limiting block diagram of a graphical user interface that may be used in implementing some or all of the methods or techniques disclosed herein, according to various embodiments described herein. [Figure 33] 1 illustrates an exemplary, non-limiting block diagram of a computing device capable of implementing some or all of the methods or techniques disclosed herein, in accordance with various embodiments described herein. [Figure 34] 1 illustrates an exemplary, non-limiting block diagram of a scientific instrument support system in which some or all of the methods or techniques disclosed herein may be implemented, according to various embodiments described herein. [Figure 35] 1 illustrates a block diagram of an exemplary non-limiting operating environment that can facilitate one or more embodiments described herein. [Figure 36] 1 illustrates an exemplary networking environment operable to perform various implementations described herein. [Figure 37] 1 illustrates an exemplary dual beam microscope that may be implemented in accordance with various embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0007] The following detailed description is merely illustrative and is not intended to limit the embodiments or the application / uses of the embodiments, nor is it intended to be bound by any expressed or implied information presented in the preceding "Background" or "Summary" sections, or in the "Detailed Description" section.

[0008] One or more embodiments will now be described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a more thorough understanding of one or more embodiments. It will be apparent, however, that in various instances, one or more embodiments may be practiced without these specific details.

[0009] Various operations may be described sequentially as multiple separate actions or operations in a manner that is most helpful for understanding the subject matter disclosed herein. However, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations may be performed in an order different from the order presented. The operations described may be performed in an order different from the described embodiment. In additional embodiments, various additional operations may be performed or described operations may be omitted.

[0010] Although some elements may be referred to in the singular (e.g., a "processing device"), any suitable element may be represented by multiple instances of that element, and vice versa. For example, a set of operations described as being performed by a processing device may be implemented with different ones of the operations performed by different processing devices. As used herein, the phrase "based on" should be understood to mean "based at least in part on," unless otherwise specified.

[0011] Charged particle microscopes (e.g., scanning electron microscopes) electron microscope (SEM), transmission electron microscope microscope (TEM), electron energy-loss microscope A charged particle microscope (EELM) can be any suitable computerized device capable of capturing or generating submicroscopic or nanoscale images or energy spectra in a scientific, laboratory, research, or clinical operating environment. To facilitate the capture or generation of such images or energy spectra, a charged particle microscope can utilize a complex arrangement of actuatable components (e.g., ion sources, electron sources, optical lenses or apertures, optical plates or deflectors, columns, coils, heaters, coolers, fluid valves, fluid pumps, circuit switches, sample stages), sensors (e.g., ion detectors, electron detectors, voltmeters, thermistors, potentiometers, pressure gauges), or consumables (e.g., carrier fluids, calibrants, filters, reactive gases).

[0012] The technical field of charged particle microscopes has historically been limited by the complexity of their operation. In other words, because charged particle microscopes can have such complex structures, they can be proportionately complex to operate or use. Indeed, in order for a user to competently or confidently use a charged particle microscope to analyze clinical or laboratory samples, the user often requires extensive specialized training, education, or certification in charged particle microscopes. For example, a user may learn how to properly operate a charged particle microscope's graphical user interface (GUI) or physical controls by taking a microscopy course for several weeks or months (after all, a charged particle microscope may have a seemingly dizzying or overwhelming number of configurable software or hardware settings, buttons, knobs, sliders, or options). Users without such extensive training may be unable to fully use a charged particle microscope. For example, a user who does not take a proper microscopy course of several weeks or months, but nevertheless attempts to operate a charged particle microscope, may be significantly more likely to damage the sample or the charged particle microscope itself.

[0013] Unfortunately, this need for extensive specialization can be significantly burdensome given the fact that different types of charged particle microscopes may perform differently from one another. That is, specialized microscopy training, education, or certification is not transferable between, within, or across different charged particle microscopes. For example, any training that prepares one to competently operate an SEM does not necessarily prepare one to competently operate a TEM. As another example, any training that prepares one to competently operate an SEM with model number A does not necessarily prepare one to competently operate an SEM with model number B. As yet another example, any training that prepares one to competently operate an SEM with model number A and software version C does not necessarily prepare one to competently operate an SEM with model number A and software version D.

[0014] In either case, such significant operational complexity may prevent a user from interacting with the charged particle microscope efficiently or intuitively, or, in other words, may be viewed as a high barrier to entry into the field of charged particle microscopy.

[0015] Therefore, systems or techniques that can reduce such entry barriers (eg, make charged particle microscopes more user-friendly or easier to operate) may be desirable.

[0016] Various embodiments described herein may address this technical problem. One or more embodiments described herein may include a system, computer-implemented method, apparatus, or computer program product that can facilitate a large-scale language model for charged particle microscope operation. In other words, various embodiments described herein may leverage a large-scale language model (LLM), such as ChatGPT, to serve as an accessible, user-friendly interface for a charged particle microscope. Thus, when various embodiments described herein are implemented, a user can interact with or otherwise operate a charged particle microscope by typing or speaking intuitive natural language commands, requests, or queries, without prior knowledge of how to expertly traverse a world of complex configurable microscope settings, buttons, or options. In other words, various embodiments described herein may be considered to reduce or eliminate the need for a user to first undergo extensive specialized training in microscopy operations before using or operating a charged particle microscope. In still other words, the various embodiments described herein may be viewed as significantly reducing the learning curve associated with charged particle microscopes or otherwise increasing the accessibility or ease of use of charged particle microscopes.

[0017] The inventors have contemplated various embodiments for achieving such enhanced accessibility or user-friendliness. As described herein, such enhanced accessibility or user-friendliness may be achieved by implementing any of the following: image or spectrum-aligned LLM monitoring of user microscopy commands, image or spectrum-aligned LLM tutorials in response to user microscopy workflow queries, image or spectrum-aligned LLM malfunction diagnosis in response to user microscopy troubleshooting queries, image or spectrum-aligned LLM explanations in response to user microscopy sample queries, or image or spectrum-aligned LLM GUI generation in response to past user microscopy queries.

[0018] Consider first the implementation of image- or spectrum-tuned LLM monitoring of user microscopy commands. In various embodiments, a user of a charged particle microscope can issue commands to the charged particle microscope (e.g., commands to perform certain microscopy tasks, commands to set certain microscopy parameters to desired values). In various aspects, the user can type or speak such commands into an appropriate human-computer interface of the charged particle microscope (e.g., type into a text field, speak into a microphone). In various instances, a given sample can be loaded into the charged particle microscope, and the execution of a command may cause unwanted or otherwise uncommon harm to the given sample. In other words, a user, due to their own inexperience or lack of expertise, may inadvertently operate the charged particle microscope in a manner that harms the given sample. Various embodiments described herein can help to ameliorate such harm. In particular, various embodiments described herein can cause a charged particle microscope to capture an image (e.g., in the case of an SEM or TEM) or an energy spectrum (e.g., in the case of an EELM) of a given sample in response to receiving a command but prior to implementing or performing the command. In various aspects, such image or spectrum capture can be facilitated according to any suitable default protocol of the charged particle microscope. In various instances, both the command and the image or energy spectrum can be provided together as a comprehensive input prompt to the LLM, thereby causing the LLM to generate a natural language response that can be visually played (e.g., on a computer screen) or audibly played (e.g., over an electronic speaker) for viewing or listening to a user. As described herein, the natural language response can textually describe or explain whether performing the command will cause unreasonable or unusual harm to a given sample.In other words, an image or energy spectrum captured by a charged particle microscope can be considered to convey at least some substantial information about a given sample (e.g., about the optical or chemical properties of the given sample), and the LLM can be considered to evaluate, judge, or otherwise double-check the appropriateness or reasonableness of a user's command based on that substantial information. In some cases, as described herein, that substantial information can be curated or otherwise enhanced by: identifying documentation related to both the command and the image or energy spectrum; obtaining inference task results (e.g., predicted or inferred classification labels) about the image or energy spectrum from an accompanying or auxiliary machine learning model; or obtaining virtual experiment results about both the command and the image or energy spectrum from a digital twin of the charged particle microscope. Indeed, such documentation, inference task results, or virtual experiment results can be provided as supplemental input to the LLM, thereby providing the LLM with even more information about the given sample. In either case, the natural language response can be viewed as a real-time warning or notification to the user as to whether or not implementing the command will damage a given sample. Thus, adjusting the LLM with an image or energy spectrum as described herein can help reduce or avoid unintended or unwanted harm to a given sample, even though the user may not have significant training, education, or certification in charged particle microscopy.

[0019] Consider next the implementation of an image- or spectrum-tuned LLM tutorial in response to a user microscopy workflow query. In various embodiments, a user of a charged particle microscope may desire to perform a particular workflow (e.g., voltage contrast analysis) using the charged particle microscope. However, the user may not know how to properly perform the particular workflow due to inexperience or lack of expertise with charged particle microscopes. Thus, in various aspects, the user may type or speak a question into any suitable human-computer interface of the charged particle microscope, which may ask how to perform the particular workflow. As described above, a given sample may be loaded into the charged particle microscope. In various cases, the particular workflow may be sample-dependent (e.g., the particular workflow may be composed of different steps or substeps depending on the type of sample on which the particular workflow is performed). Thus, similar to the above, various embodiments described herein may, in response to receiving a query, cause the charged particle microscope to capture an image or energy spectrum of the given sample via any suitable default microscopy protocol. In various aspects, both the question and the image or energy spectrum can be provided to the LLM together as a comprehensive input prompt, causing the LLM to generate a natural language response that can be visually or audibly reproduced for the user. In various cases, the natural language response can textually describe or explain a tutorial tailored to the sample for performing a particular workflow. That is, the natural language response can textually describe or explain which steps, substeps, or other actions should be performed by the user or the charged particle microscope in what order to successfully or properly perform a particular workflow on a given sample.In other words, an image or energy spectrum captured by a charged particle microscope can be considered to convey at least some substantial information about a given sample, and an LLM can be considered to utilize that substantial information to identify how to best or properly perform a particular workflow. As above, that substantial information can, in some cases, be augmented by relevant documentation, inference task results, or digital twin simulation results. In either case, the natural language response can be considered to instruct or explain to a user how to perform a particular workflow on a given sample. Thus, tailoring an LLM with an image or energy spectrum as described herein can help ensure that a user is informed in real time how to properly perform a particular workflow, even if the user does not have significant training, education, or certification in charged particle microscopy.

[0020] Consider now the implementation of image- or spectrum-tuned LLM malfunction diagnosis in response to a user microscopy troubleshooting query. In various embodiments, a given sample can be loaded into a charged particle microscope, and the charged particle microscope may experience a particular malfunction (e.g., one or more specified symptoms or error codes) when attempting to operate on the given sample. A user of the charged particle microscope may desire to troubleshoot, diagnose, or otherwise resolve the particular malfunction, but the user may not know how to do so due to inexperience or lack of expertise. Thus, in various aspects, a user can type or speak a question into any suitable human-computer interface of the charged particle microscope, and the question can be asked regarding the particular malfunction (e.g., what is causing the particular malfunction and how to fix the particular malfunction). In various cases, the user's question can describe or otherwise identify at least one known detail of the given sample (e.g., the known composition or identity of the given sample). Various embodiments can leverage such known details about a given sample to determine how to handle a particular malfunction. Indeed, similar to the above, various embodiments described herein can cause a charged particle microscope to capture an image or energy spectrum of a given sample via any suitable default microscopy protocol in response to receiving a query. In various aspects, both the query and the image or energy spectrum can be provided together as a comprehensive input prompt to the LLM, thereby causing the LLM to generate a natural language response that can be visually or audibly reproduced for the user to see or hear. In various cases, the natural language response can textually describe or explain the cause or solution for a particular malfunction.That is, the natural language response may textually describe or explain which specific setting or configuration component of the charged particle microscope is causing a particular malfunction, or may textually describe or explain which steps, substeps, or other actions should be performed in what order to successfully or appropriately handle a particular malfunction. In other words, an image or energy spectrum captured by a charged particle microscope may be considered to convey at least some measured information about a given sample, a question typed or spoken by a user may be considered to convey at least some known information about a given sample, and an LLM may be considered to compare that measured information with that known information to identify the reason or manner in which the charged particle microscope is experiencing a particular malfunction. Similarly, the measured information for a given sample may, in some cases, be augmented by related documentation, inference task results, or digital twin simulation results. In some aspects, measurement data may be further augmented by obtaining self-diagnostic test results from the charged particle microscope itself. In either case, the natural language response can be viewed as instructing or explaining to the user why a particular malfunction is occurring on a given sample, or otherwise how to prevent a particular malfunction from occurring on a given sample. Thus, adjusting the LLM with an image or energy spectrum as described herein can help ensure that a user is informed in real time how to properly resolve a particular malfunction, even if the user does not have significant training, education, or certification in charged particle microscopy.

[0021] Consider next the implementation of an image- or spectrum-tuned LLM description in response to a user microscopy sample query. In various embodiments, a given sample can be loaded into a charged particle microscope, and a user of the charged particle microscope may desire to determine some particular property or characteristic of the given sample (e.g., chemical composition, crystal structure, surface roughness). However, the user may not know how to properly utilize the charged particle microscope to determine that particular property or characteristic due to inexperience or lack of expertise. Thus, in various aspects, the user can type or speak a question into any suitable human-computer interface of the charged particle microscope, which may seek to identify the particular property or characteristic. Thus, similar to the above, various embodiments described herein can cause the charged particle microscope to capture an image or energy spectrum of the given sample via any suitable default microscopy protocol in response to receiving the query. In various aspects, both the question and the image or energy spectrum can be provided to the LLM together as a comprehensive input prompt, causing the LLM to generate a natural language response that can be visually or audibly reproduced for the user. In various cases, the natural language response can textually describe, explain, or otherwise identify specific properties or characteristics of a given sample. In other words, the image or energy spectrum captured by a charged particle microscope can be considered to convey at least some substantial information about a given sample, and the LLM can be considered to utilize that substantial information to identify or quantify specific properties or characteristics sought by the user. Similarly, the substantial information of a given sample can, in some cases, be augmented by related documentation, inference task results, or digital twin simulation results. In either case, the natural language response can be considered to identify specific properties or characteristics of a given sample as requested by the user.Thus, adjusting the LLM on an image or energy spectrum as described herein can help inform a user in real time of specific characteristics or properties, even if the user does not have significant training, education, or certification in charged particle microscopy.

[0022] Finally, consider the implementation of image- or spectrum-tuned LLM GUI creation in response to past user microscopy queries. In various embodiments, a user can load a given sample into the charged particle microscope. Additionally, in various aspects, the user can proactively ask various questions or various given commands regarding the charged particle microscope (e.g., microscopy setting commands, workflow queries, troubleshooting queries, sample queries). In various instances, the charged particle microscope may have a number of configurable software settings, buttons, knobs, sliders, or options that govern or otherwise control the operation of the charged particle microscope. In various instances, various of such configurable settings may not be relevant or applicable to a given sample. Furthermore, in various aspects, some of such configurable settings may correspond to basic microscope functionality suitable for inexperienced operators, while other of such configurable settings may correspond to advanced microscopy functionality suitable only for experienced or highly trained operators. Thus, as noted above, various embodiments described herein can cause a charged particle microscope to capture an image or energy spectrum of a given sample via any suitable default microscopy protocol. In various aspects, the image or energy spectrum and a previous query or command can be provided together as a comprehensive input prompt to the LLM, thereby causing the LLM to generate synthesized code that can be compiled, executed, or otherwise run on the charged particle microscope or any suitable computerized workstation associated with the charged particle microscope. As described herein, the synthesized code can define a GUI for the charged particle microscope that is tailored to both the given sample and the user.More specifically, an image or energy spectrum captured by a charged particle microscope can be viewed as conveying at least some substantial information about a given sample (which may be augmented with related documentation, inference task results, or digital twin simulation results), and the LLM can be viewed as utilizing that substantial information to identify which of the charged particle microscope's configurable software settings or options are relevant or irrelevant to the given sample. Thus, the GUI defined by the synthesized code can block, hide, or otherwise omit settings or options that are irrelevant to the sample. Furthermore, a user's past queries or commands can be viewed as conveying at least some information about how experienced or inexperienced the user is with charged particle microscopes, and the LLM can be viewed as utilizing that information to infer which of the charged particle microscope's configurable software settings or options are appropriate or inappropriate (e.g., too advanced for the user) for the user. Thus, the GUI defined by the synthesized code can block, hide, or otherwise omit settings or options that are inappropriate for the user. Thus, adjusting the LLM with an image or energy spectrum and with a past query or command as described herein can help ensure that the user is presented with a charged particle microscope GUI that is comparable to or appropriate for both the given sample (e.g., the GUI may vary between samples) and the user's inferred experience level (e.g., the GUI may vary depending on how much or how little microscopy training the user is inferred to have received).

[0023] Thus, different samples may require or otherwise be associated with different microscopy settings, protocols, treatments, or preparations, and various embodiments described herein enable an LLM to take such different microscopy settings, protocols, treatments, or preparations into account when synthesizing a response to a user question or command. This can be accomplished by tuning an LLM with (e.g., by having the LLM receive as input) a sample image or sample energy spectrum. An LLM tuned with such an image or spectrum can make charged particle microscopy more accessible or user-friendly (e.g., intuitively usable or operable regardless of skill or expertise).

[0024] Various embodiments described herein can be considered as computerized tools (e.g., any suitable combination of computer-executable hardware or computer-executable software) that can facilitate large-scale language model support for charged particle microscope operations. In various aspects, such computerized tools can comprise an access component, a state component, a context component, a model component, or a presenter component.

[0025] In various embodiments, there may be a charged particle microscope. In various aspects, the charged particle microscope may exhibit any suitable design or configuration (e.g., may be an SEM, a TEM, an EELM, a dual beam microscope). In various cases, the charged particle microscope may include any suitable number of configurable operational settings. In various cases, the configurable operational settings may be any suitable selectively controllable hardware or software characteristic of the charged particle microscope that may be directly adjusted or changed in response to electronic instructions or commands received from a user of the charged particle microscope (e.g., a user-controlled voltage or current setting of the charged particle microscope, a user-controlled temperature setting of the charged particle microscope, or a user-controlled actuator setting of the charged particle microscope). In various aspects, there may be any suitable sample (e.g., a semiconductor wafer or lamella) currently loaded into the charged particle microscope (e.g., currently positioned or being positioned on an operable stage of the charged particle microscope).

[0026] In various embodiments, an LLM may be present. In various aspects, the LLM may exhibit any suitable deep learning internal architecture. For example, the LLM may include any suitable number of layers of any suitable type (e.g., an input layer, one or more hidden layers, an output layer, any of which may be a convolutional layer, a dense layer, a long short-term memory (LSTM) layer, a transformer layer, a nonlinear layer, a pooling layer, a batch normalization layer, or a padding layer). As another example, the LLM may include any suitable number of neurons in various layers (e.g., different layers may have the same or different numbers of neurons). As yet another example, the LLM may include any suitable activation function (e.g., softmax, sigmoid, hyperbolic tangent, normalized linear unit) in the various neurons (e.g., different neurons may have the same or different activation functions). As yet another example, the LLM may include any suitable inter-neuron or inter-layer connections (e.g., forward connections, skip connections, recurrent connections).

[0027] Regardless of the particular internal architecture of an LLM, the LLM can be configured as a generative text-to-text model: that is, the LLM can be configured to receive as input any suitable text data (which, in various cases, may or may not be accompanied by any suitable numerical data or any suitable graphical data), and the LLM can be configured to generate as output synthesized text content (e.g., one or more synthetic sentences or sentence fragments) that is semantically or substantively based on such input text data (and, in some cases, on the accompanying numerical or graphical data).

[0028] To accomplish this, the LLM can be viewed as including an encoder portion and a synthesizer portion. In various aspects, the encoder portion can be any suitable upstream layer of the LLM configured to receive input text data (and any accompanying numerical or graphical data) and generate embeddings based on the input text data. In various instances, the synthesizer portion can be any suitable downstream layer of the LLM configured to receive those embeddings and generate synthesized text content based on those embeddings.

[0029] In various aspects, an embedding generated by an encoder portion of an LLM in response to a portion of input textual, numerical, or graphical data can be considered to be any suitable mathematical quantity (e.g., a scalar, vector, matrix, tensor, tokenization, or any suitable combination thereof) that numerically represents at least some substantive or semantic aspect of the input textual, numerical, or graphical data in a low-dimensional manner. In other words, an embedding can be smaller in size or dimensionality than such input textual, numerical, or graphical data (e.g., an order of magnitude or more smaller in some cases), but despite such smaller size, the embedding can be considered to be substantively or semantically representative of such input textual, numerical, or graphical data. In still other words, an embedding can be considered to be a latent vector representation of such input textual, numerical, or graphical data.

[0030] In either case, it may be desirable to utilize LLM to increase the accessibility or ease of operation of the charged particle microscope, and in various instances, the computerized tools described herein can accomplish this in a variety of ways.

[0031] In some embodiments, the computerized tool can utilize the LLM to increase the accessibility or ease of use of the charged particle microscope by performing automated reasonability or rationality checks on commands received by a user of the charged particle microscope.

[0032] Indeed, in various cases, there may be natural language commands associated with the charged particle microscope. In various aspects, the natural language commands may be unstructured or plain text that semantically request or command that one or more configurable operational settings of the charged particle microscope (e.g., beam voltage setting, beam current setting, stage temperature setting) be set, changed, or otherwise adjusted to one or more desired values ​​or states. In various instances, the natural language commands may be typed or spoken by a user via any suitable GUI text field or microphone of the charged particle microscope.

[0033] In various embodiments, an access component of a computerized tool can electronically access natural language instructions. For example, the access component can receive, search, or otherwise retrieve natural language instructions from any suitable centralized or distributed data structure (e.g., a graph data structure, a relational data structure, a hybrid data structure). Similarly, the access component can electronically access an LLM or charged particle microscope. For example, the access component can electronically interface or communicate with an LLM or charged particle microscope (e.g., send electronic commands to the LLM or charged particle microscope, read electronic signals from the LLM or charged particle microscope). In either case, the access component can be considered a conduit through which other components of the computerized tool can electronically interact with (e.g., read, write, edit, copy, manipulate, execute, activate, deactivate, modify) the natural language instructions, LLM, or charged particle microscope.

[0034] In various embodiments, a state component of a computerized tool, in response to receiving or accessing natural language commands, can electronically cause a charged particle microscope to scan a sample currently loaded on the charged particle microscope. In various aspects, such scanning can cause the charged particle microscope to capture an image depicting or illustrating at least a portion of the sample (e.g., a SEM scan image, a TEM scan image), or an energy spectrum of the sample (e.g., an electron energy loss spectrum) or an energy spectrum otherwise associated with the sample. In various instances, the state component can cause the charged particle microscope to perform such scanning according to any suitable default microscope protocol known or deemed to be non-destructive or non-damaging to a wide range of areas or percentages of the possible microscope sample. As a non-limiting example, a default microscopy protocol can involve using a sufficiently low default beam current (e.g., on the order of nano-amps (nA) or pico-amps (pA)) and default beam voltage (e.g., less than 5 kilovolts (kV)) so as to be known or expected not to damage, degrade, or otherwise impair all, most, or any suitable subgroup of any possible samples that the charged particle microscope is expected or designed to encounter.

[0035] In various embodiments, a model component of a computerized tool can electronically generate a natural language response by running an LLM against a natural language instruction, against an image or energy spectrum of the sample, and against a damage prompt associated with the natural language instruction. More specifically, the damage prompt can be unstructured or plain text that asks or commands the user to determine whether executing or implementing the natural language instruction will harm, damage, or otherwise damage the sample. In various aspects, the model component can concatenate the natural language instruction, the image or energy spectrum, and the damage prompt together. In various instances, the model component can feed the concatenation into an input layer of the LLM, which can complete a forward pass through one or more hidden layers of the LLM, and the output layer of the LLM can compute a natural language response based on the activations provided by the one or more hidden layers of the LLM.

[0036] In various cases, the natural language response may be synthesized text based on the natural language command and the image or energy spectrum, or that responds substantively or semantically to the damage prompt. In other words, the natural language response may be unstructured or plain text that describes or explains whether changing or adjusting one or more configurable operational settings of the charged particle microscope to one or more desired values ​​or states requested or commanded by the user will unreasonably or undesirably damage the sample. In yet other words, the image or energy spectrum may be considered to inform the LLM of at least some physical, chemical, or compositional information about the sample, and the LLM may use that information to infer or predict whether the sample will be harmed by the setting change requested or commanded by the user. In yet other words, the LLM may be considered to utilize the image or energy spectrum of the currently loaded sample to monitor or double-check the reasonableness or coherence of the natural language command, and the natural language response may be considered a conclusion or determination of that monitoring or double-check.

[0037] Here, in some cases, the level of precision, completeness, or amount of specificity or detail exhibited by the natural language response may be increased or otherwise improved by enabling the LLM to consider supplemental or contextual information about or otherwise derived from the image or energy spectrum. In various embodiments, a context component of the computerized tool may electronically obtain, collect, or otherwise access such supplemental or contextual information.

[0038] As a non-limiting example, there may be a set of auxiliary or ancillary machine learning models that can be configured to perform respective inference tasks on an input image or an input energy spectrum. As some non-limiting examples, any of such auxiliary or ancillary machine learning models may be pre-trained to perform the following: image classification or energy spectrum classification, image segmentation or energy spectrum segmentation, or image regression or energy spectrum regression. Thus, the context component can execute each machine learning model of the set of auxiliary or ancillary machine learning models on the image or energy spectrum of a currently loaded sample, and such execution can produce multiple inference task results related to the sample (e.g., multiple predicted or inferred classification labels, segmentation masks, or regression outputs). Thus, in various aspects, the model component can concatenate the multiple inference task results together with natural language instructions, images or energy spectra, and damage prompts, and the model component can generate a natural language response by running an LLM on the augmented concatenation. In various instances, multiple inference task results can be viewed as providing the LLM with deeper or richer information about the sample, which can enable the LLM to make its natural language response more accurate or detailed.

[0039] As another non-limiting example, there may be a document repository containing a plurality of documents. In various instances, each of the plurality of documents may be any suitable electronic file (e.g., a word-doc file, a portable document format (PDF) file, a webpage file) that can textually (or in some cases graphically or numerically) describe, explain, or otherwise present any suitable technical information regarding the physical, chemical, or optical properties of any suitable sample, or regarding the design, construction, operation, maintenance, or troubleshooting of any suitable charged particle microscope (e.g., the various documents may be service manuals or technical handbooks (or portions thereof) for several respective charged particle microscopes, or the various documents may be composition reports or lookup tables for several respective laboratory samples). In various instances, any of the plurality of documents may be or has been written (e.g., via any suitable word processing software, computer-aided design software, or quantitative analysis software) by a technician or engineer tasked with designing, developing, prototyping, revising, manufacturing, or investigating any suitable charged particle microscope, or any suitable sample that can be analyzed by any suitable charged particle microscope. In some cases, any document may be of or otherwise have any suitable length or size (e.g., it may be one or a few pages long, tens of pages long, or hundreds of pages long). In either case, the context component may electronically search a document repository for one or more documents that are substantially related to the natural language instructions and the image or energy spectrum. In some aspects, the context component may accomplish this via embedding search. For example, the encoder portion of the LLM may be leveraged to generate specific embeddings for the natural language instructions and the image or energy spectrum.The encoder portion can be utilized to generate a respective embedding for each document in the document repository, and any document whose embedding is closest or most similar to a particular embedding can be considered to be associated with the natural language instructions and the image or energy spectrum. Thus, in various aspects, the model component can concatenate those associated documents together with the natural language instructions, the image or energy spectrum, and the damage prompt, and the model component can generate a natural language response by running an LLM on that augmented concatenation. In various cases, the associated documents can be considered to provide the LLM with deeper or richer information about how the sample is known or expected to interact with the charged particle microscope, and such deeper or richer information can enable the LLM to make the natural language response more accurate or detailed.

[0040] As yet another non-limiting example, a charged particle microscope can be electronically synchronized with a digital twin. In various aspects, a digital twin can be any suitable combination of suitable mathematical or physics-based models that can numerically, computationally, or analytically predict, anticipate, or otherwise simulate how a charged particle microscope, or any suitable portion of a charged particle microscope, will respond to any given use scenario. More specifically, a digital twin can include a parametric state, a set of input variables, and a set of output variables. In various aspects, the set of input variables can be collectively considered as operands of the digital twin, the parametric states can be collectively considered as defining operators of the digital twin, and the set of output variables can be calculated or computed by mathematically applying the parametric states to the set of input variables (e.g., via any suitable mathematical function or composition thereof). Thus, the set of input variables can be assigned any numerical values ​​that define or describe any given use scenario, and applying the parametric states to the set of input variables can simulate, predict, or anticipate how the charged particle microscope will behave or respond (e.g., output variables) to that given use scenario. In various cases, any suitable synchronization techniques can be implemented to cause or otherwise ensure that the parametric state of the digital twin closely matches (e.g., within any suitable threshold margin thereof) the true physical state of the charged particle microscope. In various aspects, the LLM can generate one or more function calls to the digital twin based on the natural language instructions and the image or energy spectrum. In various instances, execution of such one or more function calls can cause the digital twin to run or conduct one or more virtual experiments involving the charged particle microscope and the sample. In various cases, the one or more virtual experiments can produce respective simulation results (e.g., respective simulated or predicted values ​​for the digital twin's output variables).Thus, in various aspects, the model component can concatenate those simulation results together with natural language instructions, images or energy spectra, and damage prompts, and the model component can generate a natural language response by running the LLM on that extended concatenation. In various cases, those simulation results can be viewed as providing the LLM with deeper or richer information about how the sample is expected or anticipated to interact with the charged particle microscope, and such deeper or richer information can enable the LLM to make the natural language response more accurate or detailed.

[0041] In either case, the LLM may generate a natural language response that may describe or explain in text whether the natural language command provided by the user will undesirably or unintentionally damage or harm, or is likely to damage or harm, the currently loaded sample.

[0042] In various embodiments, the presenter component of the computerized tool can electronically present the natural language response to the user in any suitable manner. As a non-limiting example, the presenter component can visually render the natural language response on any suitable computer screen or monitor associated with the charged particle microscope so that the user can see or read the natural language response. As another non-limiting example, the presenter component can audibly play the natural language response (e.g., via any suitable text-to-speech conversion technique) on any suitable speaker associated with the charged particle microscope so that the user can hear the natural language response. In this manner, the computerized tool can be thought of as warning or informing the user whether a user-requested or commanded setting change or adjustment will undesirably damage or harm the specimen.

[0043] In some aspects, if the LLM determines or concludes that executing or implementing the natural language instruction would harm or damage the currently loaded specimen, the natural language response may further include one or more recommended values ​​or states that may be or should be used instead of those specified in the natural language instruction. In such cases, the presenter component may electronically ignore the natural language instruction (e.g., refrain from changing or adjusting one or more configurable operational settings to one or more desired values ​​or states in violation of the natural language instruction). In such situations, the presenter component may instead electronically cause the charged particle microscope to change or adjust one or more configurable operational settings to one or more recommended values ​​or states.

[0044] In this manner, the computerized tool can be viewed as enhancing the accessibility or ease of use of charged particle microscopes by utilizing the LLM to automatically prevent a user from unintentionally or accidentally harming or damaging a currently loaded sample. Thus, even a user who is inexperienced or unfamiliar with charged particle microscopes can nevertheless use or operate the charged particle microscope without (or with significantly reduced) fear or risk of unintentional sample damage.

[0045] Here, in some other embodiments, computerized tools can leverage LLMs to increase the accessibility or ease of use of charged particle microscopes by providing sample-tailored answers to workflow questions asked by users of the charged particle microscope.

[0046] Indeed, in various cases, there may be a natural language workflow query associated with the charged particle microscope. In various aspects, the natural language workflow query may be unstructured or plain text that semantically requests or commands an explanation of how to correctly or properly perform some particular workflow on the charged particle microscope (e.g., which steps are involved in a voltage contrast workflow, which steps are involved in a bend workflow). As noted above, the natural language workflow query may be typed or spoken by a user.

[0047] In various embodiments, the state component of the computerized tool may, in response to receiving or accessing a natural language workflow query, cause the charged particle microscope to scan a currently loaded sample through any suitable default microscopy protocol, which in various aspects may cause the charged particle microscope to capture an image or energy spectrum of the sample.

[0048] In various embodiments, the model component of the computerized tool can electronically generate a natural language response by performing an LLM on the natural language workflow query and the image or energy spectrum (e.g., performing an LLM on the concatenation of the natural language workflow query and the image or energy spectrum).

[0049] In various cases, the natural language response may be synthesized text based on the image or energy spectrum and substantially or semantically responsive to the natural language workflow query. In other words, the natural language response may be unstructured or plain text that describes or explains which specific sequence of steps, substeps, or actions (as inferred or predicted by the LLM) should be performed by a user to perform a particular workflow. Some microscopy workflows may be sample-dependent. That is, a given microscopy workflow may involve a first sequence of steps or actions if the given microscopy workflow is to be performed on a first type of sample (e.g., a metal sample), and the given microscopy workflow may instead involve a second sequence of steps or actions if the given microscopy workflow is to be performed on a second type of sample (e.g., a plastic sample). For example, a metal sample may require different pre-scan cleaning or sanitization steps than a plastic sample. As another example, a plastic sample may require a sputter coating step, whereas the sputter coating step can be omitted for a metal sample. Thus, the image or energy spectrum can be considered to inform the LLM of at least some physical, chemical, or compositional information about the currently loaded sample, and the LLM can use that information to infer or predict what sequence of specific steps or actions is required to correctly or properly implement a particular workflow for the charged particle microscope on or with respect to the currently loaded sample. Furthermore, in some cases, a particular workflow may not be suitable or appropriate for the sample at all (e.g., a particular workflow may be reserved for organic samples, but the LLM may infer that the currently loaded sample is inorganic). In such situations, the natural language response can state or explain that a particular workflow is not applicable to the currently loaded sample.In either case, LLM can be thought of as leveraging images or energy spectra to provide the user with workflow guidance tailored to the sample.

[0050] As described above, the level of precision, completeness, or amount of specificity or detail exhibited by the natural language response may be increased or otherwise improved by enabling the LLM to consider supplemental or contextual information about or otherwise derived from the image or energy spectrum. Such supplemental or contextual information (e.g., related documents, auxiliary or ancillary inference task results, digital twin simulation results) may be electronically acquired, collected, or otherwise accessed by the context component. Thus, any such supplemental or contextual information may be provided as an additional input to the LLM. As explained above, this may be viewed as providing the LLM with deeper or richer information about how the charged particle microscope and the currently loaded sample are known, expected, or predicted to interact or relate to one another, and such deeper or richer information may enable the LLM to make the natural language response more accurate or detailed.

[0051] In either case, the LLM can generate a natural language response that can describe, explain, or instruct in text how to perform a particular workflow on the charged particle microscope in a manner that is appropriate or suitable for the currently loaded sample.

[0052] In various embodiments, as described above, a presenter component of a computerized tool can electronically (eg, visually or audibly) present the natural language response to the user.

[0053] In this manner, the computerized tool can be viewed as increasing the accessibility or ease of use of a charged particle microscope by automatically teaching or showing a user how to properly perform a requested workflow for a currently loaded sample using the LLM, thereby allowing a user who is inexperienced or unfamiliar with the charged particle microscope or the requested workflow to nevertheless use or operate the charged particle microscope competently.

[0054] In yet another embodiment, then, a computerized tool can utilize the LLM to increase the accessibility or usability of a charged particle microscope by providing sample-tailored diagnostic or troubleshooting explanations in response to malfunction questions asked by the user.

[0055] Indeed, in various cases, there may be a natural language malfunction query associated with the charged particle microscope. In various aspects, the natural language malfunction query may be unstructured or plain text that semantically requests or commands an explanation for why the charged particle microscope is experiencing one or more particular malfunction symptoms (e.g., why the charged particle microscope is displaying a specified error code, why the charged particle microscope is sounding an alarm or buzzer, why the charged particle microscope is making a creaking noise). In some instances, the natural language malfunction query may include at least some description of the currently loaded sample (e.g., these malfunctions occur when attempting to analyze a steel sample, these malfunctions occur when attempting to analyze a printed circuit board sample). As noted above, the natural language malfunction query may be typed or spoken by a user.

[0056] In various embodiments, the computerized tool state component, in response to receiving or accessing a natural language malfunction query, can cause the charged particle microscope to scan a currently loaded sample through any suitable default microscopy protocol, which in various aspects can cause the charged particle microscope to capture an image or energy spectrum of the sample.

[0057] In various embodiments, the model component of the computerized tool can electronically generate a natural language response by performing an LLM on the natural language malfunction query and the image or energy spectrum (e.g., performing an LLM on the concatenation of the natural language malfunction query and the image or energy spectrum).

[0058] In various cases, the natural language response may be synthesized text that is based on the image or energy spectrum and that substantially or semantically responds to the natural language malfunction query. In other words, the natural language response may be unstructured or plain text that describes or explains what (as inferred or predicted by the LLM) is causing the charged particle microscope to experience, exhibit, or suffer from one or more specific malfunctions noted by the user. In some cases, the natural language response may include unstructured or plain text that describes or explains how to resolve or correct one or more specific malfunction symptoms (as inferred or predicted by the LLM).

[0059] As noted above, the natural language malfunction query may include at least some known information about the currently loaded sample (e.g., may briefly identify or describe at least one physical, chemical, or compositional property of the sample). Furthermore, the image or energy spectrum may be considered to inform the LLM of at least some measured, detected, or inferred information about the currently loaded sample (e.g., measured, detected, or inferred physical, chemical, or compositional property). In various aspects, the LLM may be considered to utilize any discrepancies, inconsistencies, or correlations between such known sample information and such measured, detected, or inferred sample information to infer or predict why the charged particle microscope is experiencing or exhibiting one or more particular malfunction symptoms. For example, if the natural language malfunction query specifies that the currently loaded sample is a silicon wafer sample, the LLM may expect the image or energy spectrum to look or appear in a certain way (e.g., have or lack a particular visual pattern or artifact). The extent to which or in what manner the image or energy spectrum deviates from such expectations can be considered as at least partially indicating or suggesting to the LLM what is wrong with the charged particle microscope. Thus, the LLM can be considered as utilizing the image or energy spectrum to make a more informed malfunction diagnosis of the charged particle microscope, and the natural language response can be considered as that diagnosis.

[0060] As described above, the level of precision, completeness, or amount of specificity or detail exhibited by the natural language response may be increased or otherwise improved by enabling the LLM to consider supplemental or contextual information related to or otherwise derived from the image or energy spectrum of the currently loaded sample. In various embodiments, such supplemental or contextual information (e.g., related documentation, auxiliary or ancillary inference task results, digital twin simulation results) may be electronically acquired, collected, or otherwise accessed by the context component. In some cases, such supplemental or contextual information may include the results of automated self-diagnostic tests or checks that the charged particle microscope may perform on its own (e.g., in response to a function call generated by the LLM). As described above, such supplemental or contextual information may be viewed as providing the LLM with deeper or richer information regarding how the charged particle microscope and the currently loaded sample are known, expected, or predicted to interact or relate to one another, and such deeper or richer information may enable the LLM to make the natural language response more accurate or detailed.

[0061] In either case, the LLM can generate a natural language response that can describe or explain in text why the charged particle microscope is experiencing one or more specific malfunction symptoms that the user inquired about.

[0062] In various embodiments, as described above, the presenter component of the computerized tool can electronically (e.g., visually or audibly) present the natural language response to the user in any suitable manner.

[0063] In this manner, the computerized tool may be considered to increase the accessibility or usability of charged particle microscopes by using the LLM to automatically teach or show a user in real time why a charged particle microscope is experiencing a noted malfunction or how to fix such a malfunction, such that even a user who is inexperienced or unfamiliar with charged particle microscopes may nonetheless successfully address or handle a malfunction of the charged particle microscope.

[0064] Here, in yet other embodiments, a computerized tool can utilize the LLM to increase the accessibility or ease of use of a charged particle microscope by automatically answering questions asked by a user about the currently loaded sample.

[0065] Indeed, in various cases, there may be a natural language sample query associated with a charged particle microscope. In various aspects, the natural language sample query may be unstructured or plain text that semantically requests or commands the identification of any suitable information about the currently loaded sample (e.g., what the sample's physical or chemical composition is, which microscope protocol is best for the sample, etc.). As noted above, the natural language sample query may be typed or spoken by a user.

[0066] In various embodiments, the state component of the computerized tool may, in response to receiving or accessing a natural language sample query, cause the charged particle microscope to scan a currently loaded sample through any suitable default microscopy protocol, which in various aspects may cause the charged particle microscope to capture an image or energy spectrum of the sample.

[0067] In various embodiments, the model component of the computerized tool can electronically generate a natural language response by performing an LLM on the natural language sample query and the image or energy spectrum (e.g., performing an LLM on the concatenation of the natural language sample query and the image or energy spectrum).

[0068] In various cases, the natural language response may be synthesized text based on the image or energy spectrum and substantially or semantically responsive to the natural language sample query. For example, if the natural language sample query seeks to identify a particular physical or chemical attribute of the currently loaded sample, the natural language response may textually describe, explain, or identify that particular physical or chemical attribute (as inferred or predicted by the LLM). As another example, if the natural language sample query seeks to identify the best, optimal, or most preferred microscopy protocol to be used with the currently loaded sample, the natural language response may textually describe, explain, or identify that best, optimal, or most preferred microscopy protocol (as inferred or predicted by the LLM). Thus, the image or energy spectrum may be considered to inform the LLM of at least some physical, chemical, or compositional information about the currently loaded sample, and the LLM may use that information to synthesize a relevant answer to the natural language sample query. That is, the LLM may be considered to utilize the image or energy spectrum to answer sample-related questions asked by the user.

[0069] As described above, the level of precision, completeness, or amount of specificity or detail exhibited by the natural language response may be increased or otherwise improved by enabling the LLM to consider supplemental or contextual information about or otherwise derived from the image or energy spectrum of the currently loaded sample. Such supplemental or contextual information (e.g., related documentation, auxiliary or ancillary inference task results, digital twin simulation results) may be electronically acquired, collected, or otherwise accessed by the context component. As explained above, this may be viewed as providing the LLM with deeper or richer information about the currently loaded sample, which may enable the LLM to make the natural language response more accurate or detailed.

[0070] In either case, the LLM can generate natural language responses that can answer in text any question the user asks about the currently loaded sample.

[0071] In various embodiments, as described above, the presenter component of the computerized tool can electronically (e.g., visually or audibly) present the natural language response to the user in any suitable manner.

[0072] In this way, the computerized tool can be viewed as increasing the accessibility or ease of use of charged particle microscopy by automatically answering sample-related questions asked by a user in real time using the LLM, thus enabling users who are inexperienced or unfamiliar with charged particle microscopy to successfully analyze their samples.

[0073] In yet another embodiment, then, a computerized tool can leverage the LLM to increase the accessibility or ease of use of a charged particle microscope by providing a GUI that is tailored to the sample based on previous queries provided by the user.

[0074] Indeed, in various cases, there may be multiple previous natural language queries or commands that the user has previously asked. In various aspects, any of such previous natural language queries may be unstructured or plain text requests or commands, as described above (e.g., may command changes or adjustments to configurable operational settings of the charged particle microscope, may ask about the workflow of the charged particle microscope, may ask about a malfunction of the charged particle microscope, may ask about the currently loaded sample).

[0075] In various embodiments, the state component of the computerized tool can electronically command or otherwise cause the charged particle microscope to scan the currently loaded sample through any suitable default microscopy protocol, hi various aspects, such scanning can cause the charged particle microscope to capture an image or energy spectrum of the currently loaded sample.

[0076] In various embodiments, the model component of the computerized tool can electronically generate synthesized code by running the LLM against past natural language queries, against currently loaded sample images or energy spectra, and against GUI prompts, which can be unstructured or plain text that request or command user-customized and sample-customized GUIs to be configured for the charged particle microscope.

[0077] In various cases, the synthesized code may be one or more lines of computer code written in any suitable computer syntax or coding language that define or otherwise function as a programming script for a customized or adapted GUI of a charged particle microscope. In particular, a charged particle microscope may have a number of configurable software settings, buttons, knobs, sliders, or options, as described above. In some cases, different software settings may or may not be relevant to different types of samples (e.g., protocols or options suitable for organic samples may not be suitable for inorganic samples). Similarly, in some cases, some software settings of a charged particle microscope may be appropriate for an inexperienced user (e.g., basic or simple settings may be suitable for a technician unfamiliar with charged particle microscopes), while other software settings may only be appropriate for a highly skilled user (e.g., advanced or critical settings may only be suitable for a technician thoroughly familiar with charged particle microscopes).

[0078] In various cases, the image or energy spectrum can be considered to inform the LLM of at least some physical, chemical, or compositional information about the currently loaded sample, and the LLM can use that information to infer or predict which software settings are relevant to the currently loaded sample and which other software settings are not relevant to the currently loaded sample. Further, in some cases, the past natural language query can be considered to indirectly inform the LLM of the user's microscopy experience or training level, and the LLM can use that information to infer or predict which software settings are appropriate for the user and which other software settings are inappropriate for the user.

[0079] Thus, the synthesized code created by the LLM can define a GUI that encompasses or includes only those software settings that are relevant to the currently loaded sample and appropriate for the user's inferred skill level. In other words, the synthesized code created by the LLM can define a GUI that omits or excludes any software settings that are either irrelevant to the currently loaded sample or inappropriate for the user's or technician's inferred skill level. In either case, the LLM can be thought of as leveraging the image or energy spectrum of the currently loaded sample and previous natural language queries asked by the user to provide a tailored or customized GUI for a charged particle microscope.

[0080] As noted above, the level of accuracy, completeness, or amount of specificity or detail exhibited by the synthesized code may be increased or otherwise improved by enabling the LLM to consider supplemental or contextual information about or otherwise derived from the image or energy spectrum. Such supplemental or contextual information (e.g., related documentation, auxiliary or ancillary inference task results, digital twin simulation results) may be electronically acquired, collected, or otherwise accessed by the context component. As explained above, this may be viewed as providing the LLM with deeper or richer information about the currently loaded specimen and user, which may enable the LLM to make the synthesized code, and thus the customized GUI, more appropriate or suitable.

[0081] In either case, the LLM can generate synthesized code that can be defined or otherwise function as source code for the sample-customized and user-customized GUI of the charged particle microscope.

[0082] In various embodiments, a presenter component of the computerized tool can electronically compile, run, or otherwise execute the synthesized code to visually present the sample-customized and user-customized GUI of the charged particle microscope to the user on any suitable computer screen or monitor.

[0083] In this way, the computerized tool can be thought of as increasing the accessibility or usability of charged particle microscopes by using LLM to automatically generate a GUI based on both the current sample on the charged particle microscope and the inferred skill of the user attempting to use the charged particle microscope. Thus, users can operate or interact with the charged particle microscope intuitively or comfortably, regardless of the user's individual level of microscopy experience.

[0084] Various embodiments described herein can be used to solve problems using hardware or software that are highly technical in nature (e.g., to facilitate improved digital twin calibration for scientific instruments), are not abstract, and cannot be implemented as a set of mental operations by a human. Furthermore, some of the processes performed can be performed by specialized computers to perform defined operations related to the field of charged particle microscopy (e.g., electron microscopes such as SEM, TEM, or EELM, artificial neural networks such as LLM, digital twins).

[0085] For example, such defined actions may include: accessing, by a device operably coupled to the processor, a natural language instruction provided by a user of the charged particle microscope, the natural language instruction being able to request or command that a configurable operational setting of the charged particle microscope be changed to a target value; causing, by the device, in response to receiving the natural language instruction, the charged particle microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol; and running, by the device, a large-scale language model on both the natural language instruction and the image or energy spectrum of the sample, thereby generating a natural language response able to indicate whether changing the configurable operational setting to the target value will adversely affect the sample.

[0086] As another example, such defined operations may include: accessing, by a device operably coupled to the processor, a natural language query provided by a user of the charged particle microscope, the natural language query may request or instruct the charged particle microscope to identify a method for performing a microscopy workflow; causing, by the device, in response to receiving the natural language command, the charged particle microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol; and running, by the device, a large-scale language model on both the natural language query and the image or energy spectrum of the sample, thereby generating a natural language response that may describe a microscopy workflow tailored to the sample.

[0087] As yet another example, such defined operations may include: accessing, by a device operably coupled to the processor, a natural language query provided by a user of the charged particle microscope, the natural language query may request or command an identification of why the charged particle microscope is experiencing a malfunction with respect to a sample currently loaded on the stage of the charged particle microscope; causing, by the device, in response to receiving the natural language query, to capture an image or energy spectrum of the sample according to a default microscopy protocol; and running, by the device, a large-scale language model on both the natural language query and the image or energy spectrum of the sample, thereby producing a natural language response that may explain the cause of the malfunction.

[0088] As yet another example, such defined operations may include: accessing, by a device operably coupled to the processor, a natural language query provided by a user of the charged particle microscope, the natural language query may request or command identification of an attribute of a sample currently loaded on the stage of the charged particle microscope; causing, by the device, in response to receiving the natural language command, to capture an image or energy spectrum of the sample according to a default microscopy protocol; and running, by the device, a large-scale language model on both the natural language query and the image or energy spectrum of the sample, thereby generating a natural language response that may describe or indicate an attribute of the sample.

[0089] As yet another example, such defined operations may include: accessing, by a device operably coupled to the processor, a plurality of past natural language microscopy queries provided by a user of the charged particle microscope; causing, by the device, the charged particle microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol; running, by the device, a large-scale language model on both the plurality of past natural language microscopy queries and the image or energy spectrum of the sample, thereby generating synthesized code that defines a graphical user interface for the charged particle microscope that is adapted to both the sample and the user's inferred microscopy skill level; and running, by the device, the synthesized code, thereby rendering or activating the graphical user interface.

[0090] In various cases, any of such defined operations may involve synthesizing, via execution of the LLM, function calls for a digital twin synchronized with the charged particle microscope based on an image or energy spectrum of the sample, causing the digital twin to perform a virtual experiment or simulation according to those synthesized function calls, thereby producing experimental or simulation results for the sample, and providing such experimental or simulation results as supplemental input to the LLM to assist in generating any suitable natural language responses or synthesized code.

[0091] Such defined operations are inherently computerized. Indeed, charged particle microscopes (e.g., SEM, TEM, EELM, dual-beam microscopes) are highly specialized computerized devices equipped with specific computerized hardware (e.g., temperature sensors, pressure sensors, voltage sensors, ion beam emitters, electron beam emitters, focusing lenses, ion detectors, electron detectors, beam apertures, fluid valves, and actuable sample stages). Charged particle microscopes and the operations they perform cannot be performed in any reasonable or practical manner without a computer, by the human mind, or by a person using pen and paper. Furthermore, artificial neural networks (e.g., LLMs) are also inherently computerized constructs, equipped with specific software-oriented architectures (e.g., input layers, hidden layers, or output layers, any of which may be composed of trainable or untrainable internal parameters, such as convolutional layers or LSTM layers). Artificial neural networks cannot be trained or executed in any reasonable or practical manner without a computer, by the human mind, or by a person simply using pen and paper. Furthermore, a digital twin (as the word "digital" in its name suggests) is also an essentially computerized or virtual construct used to electronically predict or simulate the future behavior of a charged particle microscope. A digital twin simply cannot be facilitated or performed in any reasonable or practical way without a computer, by the human mind, or by a human using pen and paper.

[0092] Furthermore, the various embodiments described herein can incorporate various teachings related to the field of charged particle microscopy into practical applications. As explained above, charged particle microscopes are highly integrated and complex devices that often require extensive specialized training, education, or certification to learn how to operate. In fact, charged particle microscopes can have numerous configurable or selectable settings, buttons, knobs, sliders, or options that can be difficult for an inexperienced user to navigate. In other words, charged particle microscopes can be considered unfriendly, inaccessible, or otherwise difficult to operate.

[0093] Various embodiments described herein can help ameliorate this problem by implementing large-scale language model support for charged particle microscope operation. That is, various embodiments described herein can leverage LLMs to enhance the user-friendliness, accessibility, or ease of operation of a charged particle microscope. As described herein, various embodiments can accomplish such enhancements through any of the following: image- or spectrum-aligned LLM monitoring of user microscopy commands, image- or spectrum-aligned LLM tutorials in response to user microscopy workflow queries, image- or spectrum-aligned LLM malfunction diagnosis in response to user microscopy troubleshooting queries, image- or spectrum-aligned LLM explanations in response to user microscopy sample queries, or image- or spectrum-aligned LLM GUI creation in response to past user microscopy queries.

[0094] Indeed, given a microscopy command (e.g., a plain-text instruction to adjust or change configurable microscopy settings) from a user of the charged particle microscope, various embodiments can involve having the charged particle microscope automatically capture an image or energy spectrum of a currently loaded sample of the charged particle microscope and providing both the microscopy command and the image or energy spectrum as inputs to an LLM, thereby causing the LLM to generate a plain-text response describing whether performance of the microscopy command will unlikely or undesirably harm the currently loaded sample. That is, the LLM can be thought of as utilizing any physical, chemical, or compositional information about the currently loaded sample conveyed by the image or energy spectrum to infer whether the currently loaded sample is likely to be degraded by the microscopy command. Note that such physical, chemical, or compositional information about the currently loaded sample can optionally be augmented, supplemented, or accompanied by relevant documents found via embedded search, by inference task results predicted by auxiliary machine learning models, or by simulation results generated by a digital twin. In either case, the various embodiments described herein can be viewed as providing continuous or continuous real-time background monitoring of a user's commands to alert or notify the user when or if the user's commands are inappropriate for the currently loaded sample. This can be viewed as increasing the accessibility or ease of use of the charged particle microscope, allowing an inexperienced user to operate the charged particle microscope without fear of unintentionally or unintentionally damaging the sample. Equivalently, this can be viewed as protecting or guarding the sample from the inexperience of an inadequately trained user.

[0095] Furthermore, given a microscopy workflow question (e.g., a plain-text question asking how to perform a particular microscopy workflow) from a user of the charged particle microscope, various embodiments can involve having the charged particle microscope automatically capture an image or energy spectrum of a currently loaded sample on the charged particle microscope and providing both the microscopy workflow question and the image or energy spectrum as inputs to an LLM, thereby causing the LLM to generate a plain-text response that explains or instructs the user on which particular sequence of steps to perform in order to properly or acceptably apply the particular microscopy workflow to the currently loaded sample. That is, the LLM can be thought of as utilizing any physical, chemical, or compositional information about the currently loaded sample conveyed by the image or energy spectrum to infer which microscopy workflow steps should be performed on the currently loaded sample and in what order. Again, it should be noted that such physical, chemical, or compositional information about the currently loaded sample can optionally be augmented, supplemented, or accompanied by related documentation, accompanying inference task results, or digital twin simulation results. In either case, the various embodiments described herein can be viewed as providing a user with real-time, sample-adapted instruction or guidance for performing a microscopy workflow, which can be viewed as increasing the accessibility or user-friendliness of charged particle microscopes, such that even inexperienced or untrained users can competently perform workflows on a charged particle microscope.

[0096] Furthermore, when a microscope malfunction question is provided by a user of the charged particle microscope (e.g., a plain text question asking why a particular microscope malfunction is occurring), various embodiments may involve having the charged particle microscope automatically capture an image or energy spectrum of a currently loaded sample of the charged particle microscope, and providing both the microscope malfunction question and the image or energy spectrum as inputs to an LLM, thereby causing the LLM to generate a plain text response that explains or describes the likely reason, cause, or solution for the particular microscope malfunction. In particular, the microscope malfunction question may identify any suitable known attributes of the currently loaded sample, and the image or energy spectrum may be thought of as conveying at least some measured or detected attributes of the currently loaded sample, and the LLM may be thought of as comparing such known attributes with the measured / detected attributes to infer why the charged particle microscope is not operating as it should, or otherwise how the charged particle microscope can be repaired or corrected. Again, such measured / detected information of the currently loaded sample can be optionally augmented, supplemented, or accompanied by related documentation, ancillary inference task results, or digital twin simulation results. In any case, the various embodiments described herein can be viewed as providing a user with real-time troubleshooting advice for a charged particle microscope. This can be viewed as increasing the accessibility or user-friendliness of a charged particle microscope, so that even inexperienced or untrained users can competently handle malfunctions of the charged particle microscope.

[0097] Furthermore, given a microscopy sample query (e.g., a plain-text query seeking identification of some attribute of the currently loaded sample) from a user of the charged particle microscope, various embodiments can involve having the charged particle microscope automatically capture an image or energy spectrum of the currently loaded sample and providing both the microscopy sample query and the image or energy spectrum as inputs to an LLM, thereby causing the LLM to generate a plain-text response that identifies or describes the requested attribute of the currently loaded sample. That is, the LLM can be thought of as utilizing any physical, chemical, or compositional information about the currently loaded sample conveyed by the image or energy spectrum to infer the requested attribute. Again, such physical, chemical, or compositional information about the currently loaded sample can optionally be augmented, supplemented, or accompanied by related documentation, accompanying inference task results, or digital twin simulation results. In either case, various embodiments described herein can be thought of as providing a user with real-time sample characterization or analysis in response to the user's intuitive natural language query. This can be seen as increasing the accessibility or user-friendliness of charged particle microscopes, such that even inexperienced or untrained users can use the charged particle microscope to competently analyze the user's desired sample.

[0098] In some cases, given a history of microscopy queries from a user of the charged particle microscope, various embodiments can involve causing the charged particle microscope to automatically capture images or energy spectra of a currently loaded sample of the charged particle microscope, and providing both the history of microscopy queries and the images or energy spectra as inputs to an LLM, thereby causing the LLM to generate synthesized code defining a GUI for the charged particle microscope, which presents configurable microscopy settings that are relevant to the currently loaded sample and suitable for the user's inferred experience level, and hides configurable microscopy settings that are either not relevant to the currently loaded sample or not suitable for the user's inferred experience level. That is, the LLM can be thought of as utilizing any physical, chemical, or compositional information about the currently loaded sample conveyed by the image or energy spectrum to infer which configurable settings of the charged particle microscope are applicable or inapplicable to the currently loaded sample. Similarly, the LLM can be thought of as using the history of microscopy questions to infer how much or how little training or skill a user has with charged particle microscopy, thereby identifying which configurable settings of the charged particle microscope are manageable (e.g., not too advanced) or inoperable (e.g., too advanced) by the user. In either case, the LLM can use such inferred information to configure or define a GUI for the charged particle microscope that is customized for both the currently loaded sample and the user. This can be thought of as increasing the accessibility or user-friendliness of the charged particle microscope, such that any or all users of the charged particle microscope can be automatically presented with a GUI that is appropriate for the user's experience and the user's desired sample.

[0099] For at least the foregoing reasons, the various embodiments described herein can be considered to address or ameliorate various problems or shortcomings related to the accessibility of charged particle microscopes. Accordingly, the various embodiments described herein can be considered concrete and tangible technical improvements in the field of charged particle microscopes. Accordingly, the various embodiments described herein certainly qualify as useful and practical applications of computers.

[0100] Additionally, various embodiments described herein can control real-world tangible devices based on the disclosed teachings. For example, various embodiments described herein can electronically activate, deactivate, or otherwise operate real-world hardware (e.g., ion beam emitters, ion focusing lenses, carrier fluid valves / pumps) of a real-world charged particle microscope (e.g., SEM, TEM, EELM, dual beam microscope).

[0101] FIG. 1 illustrates an exemplary, non-limiting block diagram of a scientific instrument module 102 according to various embodiments described herein.

[0102] In various embodiments, the scientific instrument module 102 may be implemented by circuitry (e.g., including electrical or optical components) such as a programmed computing device. The logic of the scientific instrument module 102 may be contained on a single computing device or may be distributed across multiple computing devices that communicate with each other as needed. Examples of computing devices that may implement the scientific instrument module 102, alone or in combination, are discussed herein with reference to FIGS. 33 and 35, and examples of systems or networks of interconnected computing devices in which the scientific instrument module 102 may be implemented across one or more of the computing devices are discussed herein with reference to FIGS. 34 and 36.

[0103] The scientific instrument module 102 may include first logic 104, second logic 106, and third logic 108. As used herein, the term “logic” may include an apparatus that performs a set of operations associated with the logic. For example, any of the logic elements included in the scientific instrument module 102 may be implemented by one or more computing devices programmed with instructions that cause one or more processing devices of the computing devices to perform the associated set of operations. In particular embodiments, a logic element may include one or more non-transitory computer-readable media having instructions that, when executed by one or more processing devices of the one or more computing devices, cause the one or more computing devices to perform the associated set of operations. As used herein, the term “module” may refer to a collection of one or more logic elements that together perform the function associated with the module. Different logic elements within a module may take the same form or different forms. For example, some logic within a module may be implemented by a programmed general-purpose processing device, while other logic within the module may be implemented by an application-specific integrated circuit (ASIC). In another example, different ones of the logic elements in a module may be associated with different sets of instructions executed by one or more processing devices. A module may omit one or more of the logic elements depicted in an associated drawing; for example, a module may include a subset of the logic elements depicted in an associated drawing if that module performs a subset of the operations discussed herein with reference to that module.

[0104] In various embodiments, there may be a scientific instrument corresponding to the scientific instrument module 102. In various aspects, the scientific instrument may be any suitable computerized device capable of electronically measuring some scientifically relevant, clinically relevant, or research-related characteristic, property, or attribute of an analytical sample (e.g., of a known or unknown mixture, compound, or collection of substances). As one non-limiting example, the scientific instrument may be a scanning electron microscope. In such a case, the scientific instrument may measure or determine the surface topography of the analytical sample. As another non-limiting example, the scientific instrument may be a scanning electron microscope. In such a case, the scientific instrument may measure or determine details of the internal structure of the analytical sample. As yet another non-limiting example, the scientific instrument may be an electron energy loss microscope. In such a case, the scientific instrument may measure or determine location-by-location counts or intensities over a range of defined energy loss bins or bands for the analytical sample. As a more general non-limiting example, the scientific instrument may be any suitable type of charged particle microscope (e.g., some types of microscopes may use a beam of non-electron ions to capture images or energy spectra).

[0105] In various embodiments, the first logic 104 can access natural language commands typed or spoken by a user of the scientific instrument. In various aspects, the natural language commands can be one or more unstructured sentences that request or command setting one or more configurable operating parameters of the scientific instrument to one or more target values.

[0106] In various embodiments, the second logic 106 may involve, in response to receiving a natural language command, causing the scientific instrument to capture an image or energy spectrum of any sample currently loaded in or on the scientific instrument. In various aspects, the scientific instrument may utilize any suitable default instrument protocol to capture the image or energy spectrum (e.g., may utilize a default beam voltage or beam current known to be non-destructive for all or many different types of samples).

[0107] In various embodiments, the third logic 108 can involve executing the LLM on both the natural language instructions and an image or energy spectrum of the currently loaded sample. In various instances, such execution can cause the LLM to synthesize a natural language response that describes, explains, or otherwise indicates whether fulfillment of the natural language instructions will cause damage to the currently loaded sample (e.g., whether to alter one or more configurable operating parameters to one or more target values). That is, the image or energy spectrum can be viewed as conveying or containing at least some physical, chemical, or compositional information about the currently loaded sample, and the LLM can be viewed as utilizing that information to infer or predict whether the natural language instructions are likely to damage the currently loaded sample. In various instances, the third logic 108 can further include visually or audibly presenting the natural language response to a user (e.g., via a computer screen or computer speakers of the scientific instrument).

[0108] Thus, the scientific instrument module 102 can facilitate large-scale language model support for charged particle microscope operation (e.g., can continuously monitor user-provided commands for the scientific instrument to warn or notify the user whether such commands are likely to harm the loaded sample).

[0109] 2 is an exemplary, non-limiting flow diagram of a computer-implemented method 200 according to various embodiments described herein. The operations of computer-implemented method 200 may be used in any suitable context to perform any suitable operations (e.g., performed by or used in conjunction with any of the various modules, computing devices, or graphical user interfaces described with respect to FIGS. 1, 32, 33, 34, 35, and 36). Although the operations are illustrated in FIG. 2 once each and in a particular order, the operations may be reordered or repeated as desired and necessary (e.g., different operations performed may be performed in parallel, as appropriate).

[0110] In various aspects, operation 202 can include performing a first operation of accessing a natural language command provided by a user of the charged particle microscope. In various instances, the natural language command can request or command that a configurable operating parameter of the charged particle microscope be set to a target value. In various instances, first logic 104 can perform or otherwise facilitate operation 202.

[0111] In various aspects, operation 204 may include performing a second operation in response to the natural language command that causes the charged particle microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol. In various instances, second logic 106 may perform or otherwise facilitate operation 204.

[0112] In various cases, operation 206 can include performing a third operation of running a large-scale language model on both the natural language instructions and the image or energy spectrum of the sample, which can produce a natural language response indicating whether setting the configurable operational parameters to target values ​​will adversely affect the sample.

[0113] Thus, the computer-implemented method 200 can facilitate large-scale language model support for charged particle microscope operations.

[0114] FIG. 3 illustrates a block diagram of an exemplary, non-limiting system that can facilitate large-scale language model assistance for charged particle microscopy operations, in accordance with one or more embodiments described herein.

[0115] In various embodiments, a charged particle microscope 302 may be present. In various aspects, the charged particle microscope 302 may be as described above. That is, the charged particle microscope 302 may be any suitable computerized device capable of electronically capturing any suitable image of any suitable analytical sample, or any suitable energy spectrum of any suitable analytical sample, utilizing its constituent hardware (e.g., electron source, anode, condenser lens, condenser aperture, scan coil, objective lens, objective aperture, deflector, condenser, stigmator, electron detector, X-ray detector, actuable sample stage). As a non-limiting example, the charged particle microscope 302 may be any suitable SEM. As another non-limiting example, the charged particle microscope 302 may be any suitable TEM. As yet another non-limiting example, the charged particle microscope 302 may be any suitable scanning-transmission electron microscope (STEM). As yet another non-limiting example, the charged particle microscope 302 may be any suitable EELM. As yet another non-limiting example, charged particle microscope 302 can be any suitable dual beam microscope.

[0116] Although not explicitly shown in the figures, the charged particle microscope 302 may be electronically integrated with any suitable human-computer interface device, which may be remote or local from the charged particle microscope 302. Thus, a user or technician associated with the charged particle microscope 302 may interact with or otherwise control the charged particle microscope 302. Some non-limiting examples of human-computer interface devices may be a keyboard of the charged particle microscope 302, a keypad of the charged particle microscope 302, a touchscreen of the charged particle microscope 302, or a voice command system of the charged particle microscope 302.

[0117] In either case, the charged particle microscope 302 can include a plurality of configurable operational settings 304. In various aspects, each of the plurality of configurable operational settings 304 can guide, influence, or otherwise determine how the charged particle microscope 302 runs, operates, or functions with respect to any given analytical sample, and can be any suitable hardware- or software-related characteristic of the charged particle microscope 302 that can be selectively controlled, changed, adjusted, or otherwise set by a user or technician (e.g., via interaction with a human-computer interface device of the charged particle microscope 302). As a non-limiting example, any of the plurality of configurable operational settings 304 can be a user-controllable voltage setting (e.g., beam voltage) or current setting (e.g., beam current) that can enable a user or technician to selectively control electrodes of the charged particle microscope 302 to selectively increase or decrease voltages or currents applied within or by the charged particle microscope 302. As another non-limiting example, any of the plurality of configurable operation settings 304 may be a user-configurable temperature setting that may enable a user or technician to control a heater (e.g., a stage heater, a heating coil) or a cooler (e.g., a cooling fan, a heat pump, a refrigerator) of the charged particle microscope 302 to selectively increase or decrease the temperature within or applied by the charged particle microscope 302. As yet another non-limiting example, any of the plurality of configurable operation settings 304 may be a user-configurable mechanical actuator setting that may enable a user or technician to control a mechanical actuator (e.g., an electric motor, a sample stage, an aperture, a fluid pump, or a syringe) of the charged particle microscope 302 to selectively move the mechanical actuator.As yet another non-limiting example, any of the plurality of configurable operational settings 304 may be a user-configurable optical system setting that may enable a user or technician to control optical elements (e.g., optical lenses, optical deflectors) of the charged particle microscope 302 to selectively change the optical qualities (e.g., focal spot size or position, astigmatism, defocus) applied by the charged particle microscope 302.

[0118] In various aspects, the charged particle microscope 302 may have or otherwise be associated with a currently loaded sample 306. In various instances, the currently loaded sample 306 may be (as the name suggests) one currently loaded on or within the charged particle microscope 302. As a non-limiting example, the currently loaded sample 306 may be currently positioned on, located on, or otherwise affixed to an operable stage of the charged particle microscope 302 such that the currently loaded sample 306 is analyzable or scanable by the charged particle microscope 302. In various instances, the currently loaded sample 306 may be any suitable type of medical, clinical, scientific, or laboratory specimen or sample capable of exhibiting any suitable physical, chemical, compositional, or other properties, attributes, or characteristics. As a non-limiting example, the currently loaded sample 306 may be a thin section taken from a semiconductor substrate or wafer. As another non-limiting example, the currently loaded specimen 306 may be a sample or fragment taken from a defective load-bearing structure.

[0119] In various embodiments, natural language instructions 308 may be present. In various aspects, the natural language instructions 308 may be any suitable number of plain text or unstructured sentences or sentence fragments requesting (e.g., as in the case of a question) or commanding (e.g., as in the case of a statement) that one or more of the plurality of configurable operational settings 304 be set to one or more target values ​​or states 332, respectively. In various instances, the one or more target values ​​or states 332 may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, one or more categories, or any suitable combination thereof, where a user or technician desires or wishes each of the plurality of configurable operational settings 304 to be changed or adjusted. As some non-limiting examples, the natural language instructions 308 may be any of the following: "Set the beam voltage to 30 kV," "Change the beam current to 0.2 nA," or "How about increasing the sample stage temperature by 10%?"

[0120] It should be understood and appreciated that the natural language instructions 308 may not explicitly request or command that certain ones of the plurality of configurable operational settings 304 be changed to one or more target values ​​or states 332, but may instead, in some cases, request or command that the charged particle microscope 302 perform, activate, or initiate one or more specified microscope actions or tasks. As some non-limiting examples, the natural language instructions 308 may be any of the following: "Load inorganic protocol suite," "Start voltage contrast analysis," or "Begin milling." For ease of illustration and explanation, the remainder of this disclosure describes various embodiments in which the natural language instructions 308 include one or more target values ​​or states 332. However, it should nevertheless be appreciated that any of such embodiments are equally applicable or extendable to situations in which the natural language instructions 308 do not explicitly enumerate one or more target values ​​or states 332, but instead request or command the initiation or performance of a microscopy task, action, or function.

[0121] In various instances, the natural language instructions 308 may be provided or entered by a user or technician of the charged particle microscope 302 via any suitable human-computer interface device associated with the charged particle microscope 302. As a non-limiting example, the user or technician can type (e.g., via a keyboard, keypad, or touchscreen) the natural language instructions 308 into any suitable GUI text field of the charged particle microscope 302 or of a computerized workstation supporting the charged particle microscope 302 or paired with the charged particle microscope 102. As another non-limiting example, the user or technician can verbally speak into any suitable microphone of the charged particle microscope 302 or of a computerized workstation supporting the charged particle microscope 302 or paired with the charged particle microscope 102, and any suitable speech-to-text transcription system, service, or technique can convert the user's or technician's spoken words into natural language instructions 308.

[0122] In various embodiments, there may be a large-scale language model 310 (hereinafter “LLM 310”). In various aspects, the LLM 310 may comprise an encoder portion 312 and a synthesizer portion 314. In various cases, the encoder portion 312 may be considered to be upstream from the synthesizer portion 314. Similarly, the synthesizer portion 314 may be considered to be downstream from the encoder portion 312.

[0123] In various aspects, the encoder portion 312 can exhibit any suitable deep learning internal architecture. Indeed, in various cases, the encoder portion 312 can have an input layer, one or more hidden layers, and an output layer. In various cases, any of these layers can be connected together by any suitable inter-neuron or inter-layer connections, such as forward connections, skip connections, or recurrent connections. Furthermore, in various cases, any of these layers can be any suitable type of neural network layer having any suitable learnable or trainable internal parameters. For example, any of these input layers, one or more hidden layers, or output layers can be convolutional layers whose learnable or trainable parameters can be convolution kernels. As another example, any of these input layers, one or more hidden layers, or output layers can be dense layers whose learnable or trainable parameters can be weight matrices or bias values. As yet another example, any of these input layers, one or more hidden layers, or output layers can be batch normalization layers whose learnable or trainable parameters can be shift factors or scale factors. As yet another example, any of such input layers, one or more hidden layers, or output layers may be an LSTM layer whose learnable or trainable parameters may be an input state weight matrix or a hidden state weight matrix. As yet another example, any of such input layers, one or more hidden layers, or output layers may be a Transformer layer whose learnable or trainable parameters may be a single-head or multi-head attention block or other weight matrix. Furthermore, in various cases, any of such layers may be any suitable type of neural network layer having any suitable fixed or non-trainable internal parameters. For example, any of such input layers, one or more hidden layers, or output layers may be a nonlinear layer, a padding layer, a pooling layer, or a concatenation layer.

[0124] Similarly, in various instances, the synthesizer portion 314 can exhibit any suitable deep learning internal architecture. Indeed, in various instances, the synthesizer portion 314 can have an input layer, one or more hidden layers, and an output layer. In various instances, any of such layers can be coupled together by any suitable inter-neuron or inter-layer connections (e.g., forward connections, skip connections, or recurrent connections). Furthermore, in various instances, any of such layers can be any suitable type of neural network layer having any suitable learnable or trainable internal parameters (e.g., any of such input layer, one or more hidden layers, or output layer can be a convolutional layer, a dense layer, a batch normalization layer, an LSTM layer, or a transformer layer). Furthermore, in various instances, any of such layers can be any suitable type of neural network layer having any suitable fixed or non-trainable internal parameters (e.g., any of such input layer, one or more hidden layers, or output layer can be a nonlinear layer, a padding layer, a pooling layer, or a concatenation layer).

[0125] Regardless of the particular internal architecture (e.g., the particular number, type, or organization of layers) implemented within encoder portion 312, encoder portion 312 may be configured to receive text data (which may involve any suitable numerical or graphical data) and to generate embeddings based on such input text data. In contrast, regardless of the particular internal architecture implemented within synthesizer portion 314, synthesizer portion 314 may be configured to receive embeddings generated by encoder portion 312 and to generate synthesized text content based on such embeddings. As some non-limiting examples, LLM 310 may be any of the following: ChatGPT, Generative AI, Ollama, Bard, or Claude.

[0126] In various embodiments, the system 316 may be electronically integrated (e.g., via any suitable wired or wireless electronic connection) with the charged particle microscope 302, the natural language instructions 308, or the LLM 310. As described herein, the system 316 may utilize the LLM 310 and the charged particle microscope 302 to monitor or check the rationality or reasonableness of the natural language instructions 308.

[0127] In various aspects, the system 316 may comprise a processor 318 (e.g., a computer processing unit, a microprocessor) and non-transitory computer-readable memory 320 operatively, operably, or communicatively connected or coupled to the processor 318. The non-transitory computer-readable memory 320 may store computer-executable instructions that, when executed by the processor 318, cause the processor 318 or other components of the system 316 (e.g., the access component 322, the state component 324, the context component 326, the model component 328, the presenter component 330) to perform one or more operations. In various embodiments, the non-transitory computer-readable memory 320 may store, and the processor 318 may execute, the computer-executable components (e.g., the access component 322, the state component 324, the context component 326, the model component 328, the presenter component 330).

[0128] In various embodiments, the system 316 can include an access component 322. In various aspects, the access component 322 can electronically access the charged particle microscope 302 or the LLM 310. That is, the access component 322 can electronically communicate with or interact with (e.g., send electronic instructions or commands to, receive electronic data from) the charged particle microscope 302 or the LLM 310. Thus, the access component 322 can be considered a proxy or conduit through which other components of the system 316 can interact with, communicate with, or otherwise manipulate the charged particle microscope 302 or the LLM 310. In various instances, the access component 322 can electronically access the natural language instructions 308. That is, the access component 322 can electronically receive, retrieve, or otherwise obtain the natural language instructions 308 from any suitable electronic source or database (e.g., from the charged particle microscope 302 or from an associated computerized workstation, as the case may be). In either case, the access component 322 can be considered a proxy or conduit through which other components of the system 316 interact with, control, or otherwise manipulate the natural language instructions 308. However, these are merely non-limiting examples. In other cases, the access component 322 can be omitted, and any other component of the system 316 can communicate or interact directly with the charged particle microscope 302, the LLM 310, or the natural language instructions 308.

[0129] In various embodiments, the system 316 can include a state component 324. In various aspects, the state component 324 can cause the charged particle microscope 302 to capture an image or energy spectrum of the currently loaded sample 306, as described herein. In some cases, the state component 324 can also obtain the current state of health of the charged particle microscope 302 from a digital twin associated with the charged particle microscope 302.

[0130] In various embodiments, the system 316 can include a context component 326. In various instances, the context component 326 can identify various supplemental information that can provide valuable or enriching context to the natural language instructions 308, to the currently loaded image or energy spectrum of the sample 306, or to the current health state of the charged particle microscope 302, as described herein.

[0131] In various embodiments, the system 316 can include a model component 328. In various cases, the model component 328 can execute the LLM 310, as described herein, against the natural language instructions 308, against an image or energy spectrum of the currently loaded sample 306, against the current health state of the charged particle microscope 302, or against any supplemental data obtained by the context component 326. In various aspects, such execution can produce a natural language response that can explain or describe whether the performance or implementation of the natural language instructions 308 will undesirably harm the currently loaded sample 306.

[0132] In various embodiments, the system 316 can include a presenter component 330. In various instances, the presenter component 330 can visually or audibly render natural language responses as described herein to be seen or heard by a user or technician of the charged particle microscope 302.

[0133] It should be noted that in various instances, the access component 322, the state component 324, the context component 326, the model component 328, and the presenter component 330 can be considered collectively to be one or more software components 321 of the system 316. It should be appreciated that in various aspects, the one or more software components 321 are described herein as including primarily five components (e.g., the access component 322, the state component 324, the model component 328, and the presenter component 330) for ease of explanation and illustration. However, the one or more software components 321 are not limited to being implemented as exactly these five components in every embodiment. Indeed, in some embodiments, the functionality described herein of these five components may be combined in any suitable manner to be implemented in or by fewer than five components (e.g., in some cases, a single component may perform all of the functionality described herein with respect to the access component 322, the state component 324, the context component 326, the model component 328, and the presenter component 330). In other embodiments, the functionality described herein for such five components may instead be distributed, separated, divided, or fragmented in any suitable manner to be implemented in or by more than five components (e.g., two or more components may facilitate the functionality performable by the access component 322, two or more components may facilitate the functionality performable by the state component 324, two or more components may facilitate the functionality performable by the context component 326, two or more components may facilitate the functionality performable by the model component 328, and two or more components may facilitate the functionality performable by the presenter component 330).

[0134] FIG. 4 illustrates a block diagram of an exemplary, non-limiting system including a sample image, a sample energy spectrum, and a current microscope health state that can facilitate large-scale language model assistance for charged particle microscope operation in accordance with one or more embodiments described herein.

[0135] In various embodiments, the status component 324 can electronically collect various information regarding the current or current state of the charged particle microscope 302 in response to electronic receipt or electronic access of the natural language command 308. In various aspects, such information can include an image 402, an energy spectrum 404, or a current state of health 406. Various non-limiting aspects are described with respect to FIGS.

[0136] FIG. 5 illustrates an exemplary, non-limiting block diagram showing how an image 402 or an energy spectrum 404 may be obtained according to one or more embodiments described herein.

[0137] In various embodiments, the state component 324 can, in response to receiving or accessing the natural language command 308, cause, command, or otherwise instruct the charged particle microscope 302 to electronically scan, scan, or otherwise instruct the charged particle microscope 302 to scan the currently loaded sample 306. In various aspects, such scanning can be performed by the charged particle microscope 302 and can follow any suitable default microscopy protocol known or expected to be non-destructive for all, most, many, or any suitable subset of any sample population that the charged particle microscope 302 may potentially encounter in a clinical, scientific, or laboratory setting. In other words, the default microscopy protocol can be any suitable microscopy scan in which any of the multiple configurable operational settings 304 are set, changed, or adjusted to any suitable default value or state known or expected to be safe for a wide range of potential or possible samples. As some non-limiting examples, a default microscopy protocol may involve the beam current settings, beam voltage settings, and stage temperature settings of the charged particle microscope 302 being set, changed, or adjusted to any suitable default amperage, voltage, and temperature values ​​that are known or expected not to damage, impair, or wear out most potential samples.For example, the beam voltage, beam current, and stage temperature of the default microscopy protocol may be set, changed, or adjusted to any low threshold or otherwise non-extreme value that is widely recognized as not harming many samples (e.g., if it is known or expected that most or many different types of samples will not be harmed by a beam voltage of less than 10 kV, the default microscopy protocol may have the beam voltage setting set to any suitable value less than 10 kV; if it is known or expected that most or many different types of samples will not be harmed by a beam current of less than 0.1 nA, the default microscopy protocol may have the beam current setting set to any suitable value less than 0.1 nA; if it is known or expected that most or many different types of samples will not be harmed by a stage temperature of 300 Kelvin (K), the default microscopy protocol may have the stage temperature setting set to 300 K). In some aspects, such a scan may produce an image 402. In other aspects, such a scan may produce an energy spectrum 404.

[0138] In various instances, the image 402 may visually depict or illustrate the currently loaded sample 306 or any portion thereof in any suitable manner. In some cases, the image 402 may be an x×y array of pixels, for any suitable positive integers x and y. In other cases, the image 402 may be an x×y×z array of voxels, for any suitable positive integers x, y, and z. In various aspects, the visual quality or appearance (e.g., brightness, contrast, resolution, color) of the image 402 may vary with or otherwise depend on a default microscopy protocol implemented by the charged particle microscope 302 (e.g., may depend on default values ​​or states of multiple configurable operational settings 304 that the charged particle microscope 302 uses to scan the currently loaded sample 306). As a non-limiting example, the default microscopy protocol may be any suitable type of backscattered electron detection (BSE) scanning. In such a case, the image 402 can be considered to capture, convey, or otherwise represent various crystallographic, topographic, or magnetic field information about the currently loaded sample 306. As another non-limiting example, the default microscopy protocol can be any suitable type of electron backscatter diffraction (EBSD) scan. In such a case, the image 402 can be considered to capture, convey, or otherwise represent various crystal structure or orientation information about the currently loaded sample 306. As yet another non-limiting example, the default microscopy protocol can be any suitable cathodoluminescence scan. In such a case, the image 402 can be considered to capture, convey, or otherwise represent high-resolution topographic information about luminescent portions (if any) of the currently loaded sample 306.

[0139] In various embodiments, the energy spectrum 404 can be considered a plot of measured intensity versus energy (or energy loss). As a non-limiting example, the energy spectrum 404 can be any suitable graph in which the horizontal axis represents or can span any suitable number of electron energy bins or bands (e.g., measured in eV) and the vertical axis represents or can show detected electron counts (e.g., measured in electrons or arbitrary units (AU)). In other words, the charged particle microscope 302 can irradiate the currently loaded sample 306 with an electron beam (or any other suitable charged particle beam), and these irradiated electrons can interact with the currently loaded sample 306 (e.g., pass through, bounce back, or emit X-rays during or after collision therewith), and the unique chemical composition of the currently loaded sample 306 can cause different percentages of these irradiated electrons to interact with the currently loaded sample 306 according to different energy levels, and the energy spectrum 404 can be considered to indicate how many electrons at each defined energy level are detected to have interacted with the currently loaded sample 306. As noted above, the visual quality or appearance of the energy spectrum 404 can vary with or otherwise depend on the default microscopy protocol implemented by the charged particle microscope 302. As a non-limiting example, the default microscopy protocol can be any suitable energy-dispersive X-ray spectroscopy (EDS) scan. In such a case, the energy spectrum 404 can be considered to capture, transmit, or otherwise represent the elemental composition or chemical properties of the currently loaded sample 306. As another non-limiting example, the default microscopy protocol can be any suitable electron energy-loss (EEL) scan. In such a case, the energy spectrum 404 can be considered to capture, transmit, or otherwise represent the atomic cross-section or atomic concentrations of the currently loaded sample 306.

[0140] In some embodiments, the state component 324 can cause the charged particle microscope 302 to generate only the image 402 and not the energy spectrum 404. In other embodiments, the state component 324 can cause the charged particle microscope 302 to generate only the energy spectrum 404 and not the image 402. In still other embodiments, the state component 324 can cause the charged particle microscope 302 to generate both the image 402 and the energy spectrum 404. In yet other embodiments, the state component 324 can cause the charged particle microscope 302 to generate multiple instances of the image 402 (e.g., one image using the default BSE scan, another image using the default EBSD scan, and yet another image using the default cathodoluminescence scan) or multiple instances of the energy spectrum 404 (e.g., one energy spectrum using the default EDS scan and another energy spectrum using the default EEL scan).

[0141] In either case, the image 402 or energy spectrum 404 can be considered to represent, contain, or convey at least some information regarding any suitable physical, chemical, or compositional attributes, properties, or characteristics of the currently loaded sample 306.

[0142] FIG. 6 illustrates an exemplary, non-limiting block diagram showing how the current health state 406 may be obtained according to one or more embodiments described herein.

[0143] In various embodiments, the charged particle microscope 302 can be electronically synchronized with the digital twin 602. In various aspects, the digital twin 602 can include a parametric state 604, a set of input variables 606, and a set of output variables 608.

[0144] In various aspects, the digital twin 602 may be any suitable collection or set of any suitable mathematical or physics-based models that collectively can simulate, predict, or otherwise forecast any suitable details or aspects of the behavior of the charged particle microscope 302. As a non-limiting example, the digital twin 602 may include any suitable mass continuity equation, inequality, or expression related in any way to the charged particle microscope 302. As another non-limiting example, the digital twin 602 may include any suitable energy balance equation, inequality, or expression related in any way to the charged particle microscope 302. As yet another non-limiting example, the digital twin 602 may include any suitable heat transfer equation, inequality, or expression related in any way to the charged particle microscope 302. As yet another non-limiting example, the digital twin 602 may include any suitable fluid flow equation, inequality, or expression related in any way to the charged particle microscope 302. As yet another non-limiting example, digital twin 602 may include any suitable motion or kinematic equation, inequality, or formula related in any way to charged particle microscope 302. As another non-limiting example, digital twin 602 may include any suitable Newtonian or quantum mechanical equation, inequality, or formula related in any way to charged particle microscope 302. As yet another non-limiting example, digital twin 602 may include any suitable corrosion or degradation equation, inequality, or formula related in any way to charged particle microscope 302.

[0145] In either case, the digital twin 602 can be thought of as a collection or set of mathematical or physics-based models that can simulate, predict, or forecast something about the charged particle microscope 302, and those mathematical or physics-based models can be thought of as consisting of parametric states 604, a set of input variables 606, and a set of output variables 608. In particular, the parametric states 604 can be thought of as defining the operators or coefficients of those mathematical or physics-based models, the set of input variables 606 can be thought of as operands or arguments of those mathematical or physics-based models, and the set of output variables 608 can be thought of as simulated, predicted, or expected results calculated by those mathematical or physics-based models.

[0146] In various aspects, the parametric state 604 of the digital twin 602 may include s parameters, i.e., parameter 604(1) through parameter 604(s), for any suitable positive integer s. In various cases, each parameter of the parametric state 604 may be any suitable mathematical quantity capable of representing a respective physical or theoretical property, attribute, or property of the charged particle microscope 302. For example, parameter 604(1) may be a scalar, vector, matrix, tensor, or any suitable combination thereof capable of representing a first physical or theoretical property, attribute, or property of the charged particle microscope 302. Similarly, parameter 604(s) may be a scalar, vector, matrix, tensor, or any suitable combination thereof capable of representing an sth physical or theoretical property, attribute, or property of the charged particle microscope 302.As some non-limiting examples, any parameter of the parametric state 604 may be a length of the charged particle microscope 302 or any part or component thereof, a width of the charged particle microscope 302 or any part or component thereof, a height of the charged particle microscope 302 or any part or component thereof, a thickness of the charged particle microscope 302 or any part or component thereof, a radius of curvature of the charged particle microscope 302 or any part or component thereof, a mass or density of the charged particle microscope 302 or any part or component thereof, a stiffness of the charged particle microscope 302 or any part or component thereof, an attenuation coefficient of the charged particle microscope 302 or any part or component thereof, a density of the charged particle microscope 302 or any part or component thereof, a The parameter may represent the electrical resistance of a component of the charged particle microscope 302, the electrical impedance of the charged particle microscope 302 or any part or component thereof, the thermal resistance of the charged particle microscope 302 or any part or component thereof, the thermal conductivity of the charged particle microscope 302 or any part or component thereof, the heat capacity of the charged particle microscope 302 or any part or component thereof, the optical opacity of the charged particle microscope 302 or any part or component thereof, the optical aberration coefficients (e.g., defocus coefficient, double astigmatism coefficient) of the charged particle microscope 302 or any part or component thereof, the decoherence time of the charged particle microscope 302 or any part or component thereof, or the quantum Hamiltonian element of the charged particle microscope 302 or any part or component thereof.

[0147] In some cases, any parameter of the parametric state 604 may represent a physical or theoretical property, attribute, or characteristic that is expected to be non-transient, constant, or fixed. That is, the value of such physical or theoretical property, attribute, or characteristic may not be expected to drift, decay, or otherwise change over time or with use of the charged particle microscope 302. However, in other cases, any parameter of the parametric state 604 may represent a physical or theoretical property, attribute, or characteristic that is expected to be transient, not fixed, or not constant. That is, the value of such physical or theoretical property, attribute, or characteristic may be expected to drift, decay, or otherwise change slowly or quickly over time or with use of the charged particle microscope 302.

[0148] It should be noted that in various aspects, there may be cases where any parameter of the parametric state 604 is not directly or explicitly controlled or selected by any of the plurality of configurable operational settings 304. Despite this lack of direct and explicit control, there may still be cases where any parameter of the parametric state 604 can be indirectly influenced or changed by one or more of the plurality of configurable operational settings 304. As a non-limiting example, assume that the parametric state 604 includes aberration coefficient parameters. Although aberration coefficients can be considered useful theoretical properties that help quantitatively describe the optical performance or behavior of the charged particle microscope 302, the charged particle microscope 302 may not have an aberration coefficient knob, slider, joystick, physical button, or software button that allows for explicit, direct selection of a particular desired aberration coefficient value. However, even so, the configurable knobs, sliders, joysticks, physical buttons, or software buttons that the charged particle microscope actually has (e.g., defocus knob, stigmator knob, sample stage actuator joystick, voltage knob, temperature knob) can nevertheless indirectly affect the aberration coefficients of the charged particle microscope 302 (e.g., adjusting any of the defocus knob, stigmator knob, sample stage actuator joystick, voltage knob, or temperature knob can increase or decrease the aberration coefficients of the charged particle microscope 302).

[0149] In various aspects, the set of input variables 606 of the digital twin 602 may include t variables, i.e., input variable 606(1) through input variable 606(t), for any suitable positive integer t. In various cases, each of the set of input variables 606 may be any suitable mathematical quantity capable of representing any suitable dimension, feature, detail, or aspect of a usage scenario that may be encountered or experienced by the charged particle microscope 302. For example, input variable 606(1) may be a scalar, vector, matrix, tensor, or any suitable combination thereof capable of representing a first dimension, feature, detail, or aspect of a usage scenario that may be encountered by the charged particle microscope 302. Similarly, input variable 606(t) may be a scalar, vector, matrix, tensor, or any suitable combination thereof capable of representing a tth dimension, feature, detail, or aspect of a usage scenario that may be encountered by the charged particle microscope 302. As some non-limiting examples, any of the set of input variables 606 may represent the mass or density of an analytical sample that the charged particle microscope 302 can analyze, the chemical composition of an analytical sample that the charged particle microscope 302 can analyze, the crystalline structure of an analytical sample that the charged particle microscope 302 can analyze, the absorption coefficient of an analytical sample that the charged particle microscope 302 can analyze, the warm-up time given or assigned to the charged particle microscope 302 for analyzing the analytical sample, the run time given or assigned to the charged particle microscope 302 for analyzing the analytical sample, the cool-down time given or assigned to the charged particle microscope 302 for analyzing the analytical sample, the maximum or minimum voltage level used by the charged particle microscope 302 for analyzing the analytical sample, the maximum or minimum radiation level used by the charged particle microscope 302 for analyzing the analytical sample, the maximum or minimum fluid flow rate used by the charged particle microscope 302 for analyzing the analytical sample, or the maximum or minimum temperature level used by the charged particle microscope 302 for analyzing the analytical sample.

[0150] In various aspects, the set of output variables 608 of digital twin 602 may include u variables, i.e., output variable 608(1) through output variable 608(u), for any suitable positive integer u. In various cases, each of the set of output variables 608 may be any suitable mathematical quantity capable of representing any suitable characteristic, attribute, property, feature, detail, or behavioral aspect of charged particle microscope 302 that is desired to be simulated, predicted, or predicted. For example, output variable 608(1) may be a scalar, vector, matrix, tensor, or any suitable combination thereof capable of representing a first simulated, predicted, or predicted characteristic, attribute, property, feature, detail, or behavioral aspect of charged particle microscope 302. Similarly, output variable 608(u) may be a scalar, vector, matrix, tensor, or any suitable combination thereof capable of representing a uth simulated, predicted, or predicted characteristic, attribute, property, feature, detail, or behavioral aspect of charged particle microscope 302. As a non-limiting example, any of the sets of output variables 608 may represent a simulated image or simulated energy spectrum that can be captured or generated by the charged particle microscope 302 in response to any given usage scenario. As another non-limiting example, any of the sets of output variables 608 may represent a total or limit amount of degradation or wear that the charged particle microscope 302, or any part or component thereof, may experience or accumulate in response to any given usage scenario. As yet another non-limiting example, any of the sets of output variables 608 may represent a total or limit amount of fuel, calibrant, or electricity that can be consumed by the charged particle microscope 302 during or in response to any given usage scenario. As yet another non-limiting example, any of the sets of output variables 608 may represent an amount of useful life (e.g., expressed in hours, runs, or scans) remaining or to be left of the charged particle microscope 302 in response to any given usage scenario.

[0151] Thus, the digital twin 602 can simulate, predict, or forecast how the charged particle microscope 302 will respond to any particular use scenario. Specifically, a set of input variables 606 may be assigned any suitable numerical values ​​that correspond to or otherwise define that particular use scenario, a parametric state 604 may be applied to the set of input variables 606 (e.g., according to any mathematical function, operation, or formula that comprises the digital twin 602), and a set of output variables 608 may be equal to the calculated result (e.g., product, difference, sum, quotient) of such application.

[0152] In various aspects, the state component 324 may instruct or otherwise cause the digital twin 602 to return or provide the current state of health 406 in response to electronically receiving or accessing the natural language command 308. In various instances, the current state of health 406 may be any suitable electronic data calculated or tracked by the digital twin 602 and related to the current health or maintenance of the charged particle microscope 302. As non-limiting examples, the current state of health 406 may include a latest or current version of the parametric state 604, or any subset thereof (e.g., may include the current or most recently synchronized aberration coefficients of the charged particle microscope 302), or may include a most recently calculated or recorded version of the set of output variables 608, or any subset thereof (e.g., may include the current remaining useful life of the charged particle microscope 302, or any component thereof, which may include the current total or accumulated wear of the charged particle microscope 302 or any component thereof).

[0153] FIG. 7 illustrates a block diagram of an exemplary, non-limiting system including a set of relevant documents, a set of inference task results, and a set of simulation results that can facilitate large-scale language model assistance for charged particle microscopy operations in accordance with one or more embodiments described herein.

[0154] In various embodiments, the context component 326 can electronically collect various supplemental or additional information that enhances or otherwise provides context about the natural language instructions 308, the image 402, the energy spectrum 404, or the current health state 406. In various aspects, such supplemental or additional information can include a set of related documents 702, a set of inference task results 704, or a set of simulation results 706. Various non-limiting aspects are described with respect to FIGS.

[0155] 8-12 illustrate exemplary, non-limiting block diagrams that show how a set of relevant documents 702, a set of inference task results 704, and a set of simulation results 706 may be obtained in accordance with one or more embodiments described herein.

[0156] Consider first Fig. 8. In various aspects, the context component 326 can electronically execute the LLM 310 against the natural language instructions 308, against the image 402, against the energy spectrum 404, against the current health state 406, or any suitable combination thereof. In various instances, such execution can cause the LLM 310 to generate some synthesized text 804.

[0157] More specifically, the context component 326 may concatenate together the natural language instructions 308, the images 402, the energy spectrum 404, and the current health state 406. Note that in some cases, such concatenation may include any other combination of the foregoing (e.g., may include less than all of the natural language instructions 308, the images 402, the energy spectrum 404, and the current health state 406). In various cases, the context component 326 may feed or route its concatenation to an input layer of the encoder portion 312. In various cases, the concatenation may complete a forward pass through one or more hidden layers of the encoder portion 312. In various aspects, the output layer of the encoder portion 312 may compute or otherwise calculate the embedding 802 based on activation or feature maps provided by one or more hidden layers of the encoder portion 312.

[0158] In various instances, the embedding 802 can be considered a latent vector representation that the encoder portion 312 considers or infers corresponds to a concatenation of the natural language instructions 308, the image 402, the energy spectrum 404, or the current health state 406. More specifically, the embedding 802 can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, or any suitable combination thereof. In various aspects, the dimensionality of the embedding 802 (e.g., the total number or cardinality of numerical elements in the embedding 802) can be less (e.g., in some cases, orders of magnitude less) than the total or cumulative dimensionality of the natural language instructions 308, the image 402, the energy spectrum 404, or the current health state 406. In various instances, even if the embedding 802 has a lower dimensionality, the embedding 802 can nevertheless be considered to represent, albeit in a hidden or unseen manner, at least some substantive or semantic content of the natural language instructions 308, the image 402, the energy spectrum 404, or the current health state 406. In other words, the embedding 802 can be considered a compact or compressed numerical representation of the natural language instructions 308, the image 402, the energy spectrum 404, or the current state of health 406. It should be noted that because a third party without a connection or relationship with the encoder portion 312 cannot recreate or infer the natural language instructions 308, the image 402, the energy spectrum 404, or the current state of health 406 from the embedding 802 alone, the embedding 802 can be considered to represent the natural language instructions 308, the image 402, the energy spectrum 404, or the current state of health 406 in an implicit, ambiguous, or otherwise hidden manner.

[0159] Here, in various aspects, the embedding 802 may be fed or routed to an input layer of the synthesizer portion 314. In various cases, the embedding 802 may complete a forward pass through one or more hidden layers of the synthesizer portion 314. In various aspects, the output layer of the synthesizer portion 314 may compute or otherwise calculate the embedding 804 based on activation or feature maps provided by one or more hidden layers of the synthesizer portion 314.

[0160] In various aspects, the synthesized text 804 may be one or more declarative sentences or sentence fragments generated by the synthesizing portion 314 based on the embeddings 802. Note that the synthesized text 804 need not be an estimated or approximate reconstruction of the natural language instructions 308, the image 402, the energy spectrum 404, or the current health state 406. Instead, the synthesized text 804 may be any suitable number of synthesized sentences that are semantically or substantially related in some way to the embeddings 802 and, therefore, to the natural language instructions 308, the image 402, the energy spectrum 404, or the current health state 406. In some cases, the synthesized text 804 may be considered to include hallucinations that are semantically or substantially related to the natural language instructions 308, the image 402, the energy spectrum 404, or the current health state 406.

[0161] In various aspects, the context component 326 can ignore, discard, or delete the synthesized text 804. However, the context component 326 can record, save, store, or otherwise maintain the embedding 802. In other words, the context component 326 can extract the embedding 802 from the encoder portion 312 (e.g., from a hidden layer of the LLM 310).

[0162] Now consider FIG. 9. In various embodiments, there may be a document repository 902. In various aspects, the document repository 902 may include a plurality of documents 904. In various cases, the plurality of documents 904 may include n documents, i.e., documents 904(1) through 904(n), for any suitable positive integer n>1. In various cases, each of the plurality of documents 904 may be any suitable electronic file (e.g., a word-document file, a PDF file, a webpage file) that textually (or in some cases graphically or numerically) describes, teaches, illustrates, indicates, or otherwise conveys one or more technical features, details, or aspects of any suitable charged particle microscope (which may or may not include charged particle microscope 302) or one or more technical features, details, or aspects of any suitable microscope sample (which may or may not include currently loaded sample 306).As some non-limiting examples, any of the plurality of documents 904 may include the technical or scientific design of any suitable charged particle microscope (e.g., may list or show different components, parts, or subsystems of various microscopes and describe how those components, parts, or subsystems work), how to operate any suitable charged particle microscope (e.g., may list or show different user-configurable settings or controls of various microscopes and explain what such settings or controls do), any suitable information regarding the expected use or expected operation of any suitable charged particle microscope (e.g., may show or describe various operating conditions or use case scenarios that various microscopes are or are not designed to handle), any suitable information regarding how any suitable charged particle microscope is expected or expected to be maintained, The service manual may be a service manual, maintenance handbook, schematic diagram, failure mode report, or any portion thereof that describes or explains suitable information (e.g., it may list, display, or describe various service tasks that are expected to be performed on various microscopes), any suitable information regarding troubleshooting any suitable charged particle microscope (e.g., it may describe or describe how to resolve various malfunction symptoms of various microscopes), or any suitable information regarding summarized material properties or characteristics of any suitable samples that are frequently or commonly analyzed by microscopes (e.g., it may list, describe, or describe how different types of samples should be analyzed or prepared for analysis, it may list, describe, or describe how different types of samples may be affected by different types of microscopes or by different microscopy settings).

[0163] It should be appreciated and understood that any of the plurality of documents 904 may be any sub-portion or sub-part of some larger document. For example, any of the plurality of documents 904 may be a chapter, a section, a paragraph, or even a sentence from some longer document.

[0164] In various aspects, the context component 326 can electronically generate the plurality of embeddings 906 by performing the LLM 310 as described above for each of the plurality of documents 904 .

[0165] As a non-limiting example, context component 326 can execute LLM 310 on document 904(1), and during execution, context component 326 can extract embedding 906(1) from LLM 310. More specifically, context component 326 can feed or route document 904(1) to an input layer of encoder portion 312, document 904(1) can complete a forward pass through one or more hidden layers of encoder portion 312, and the output layer of encoder portion 312 can compute or otherwise calculate embedding 906(1) based on activation or feature maps provided by the one or more hidden layers of encoder portion 312. Note that embedding 906(1) may have the same format, size, or dimensionality as embedding 802 (e.g., embeddings may be uniform-dimensional or uniform-sized vectors representing sentences, per-sentence embeddings for paragraphs of sentences may be aggregated or averaged together to produce a paragraph embedding, per-paragraph embeddings for sections or chapters may be aggregated or averaged together to produce a section or chapter embedding, and per-section or per-chapter embeddings for the entire document may be aggregated or averaged together to produce an embedding for the entire document), and thus embedding 906(1) may be considered a latent vector representation of document 904(1). In various cases, embedding 906(1) may then complete a forward pass through synthesizer portion 314, while context component 326 may ignore, discard, or remove any synthesized text content that synthesizer portion 314 creates based on embedding 906(1).

[0166] As another non-limiting example, the context component 326 can execute the LLM 310 on the document 904(n), and during execution, the context component 326 can extract an embedding 906(n) from the LLM 310. Indeed, similar to that described above, the context component 326 can feed or route the document 904(n) to the input layer of the encoder portion 312, the document 904(n) can complete a forward pass through one or more hidden layers of the encoder portion 312, and the output layer of the encoder portion 312 can compute or otherwise calculate the embedding 906(n) based on the activation map or feature map provided by the one or more hidden layers of the encoder portion 312. As such, the embedding 906(n) can have the same format, size, or dimensionality as the embedding 802, and thus the embedding 906(n) can be considered a latent vector representation of the document 904(n). As described above, the embedding 906(n) can then complete a forward pass through the synthesizer portion 314, but the context component 326 can ignore, discard, or delete any synthesized text content that the synthesizer portion 314 creates based on the embedding 906(n).

[0167] In various cases, embeddings 906(1) through 906(n) may be considered collectively as multiple embeddings 906.

[0168] In various aspects, the context component 326 can electronically determine or identify the set of relevant documents 702 by comparing the embedding 802 to the plurality of embeddings 906. In particular, for each given embedding in the plurality of embeddings 906, the context component 326 can calculate any suitable error or similarity value between the given embedding and the embedding 802. As some non-limiting examples, such error or similarity value can be calculated using a mean absolute error (MAE) calculation, a mean squared error (MSE), or a similarity value. The method may involve a mean squared error (MSE) calculation, a cosine similarity calculation, a Euclidean distance calculation, or a cross-entropy calculation. In any case, the context component 326 may conclude that the set of relevant documents 702 is any of the plurality of documents 904 whose embedding (e.g., at 906) is most similar or closest to the embedding 802. As a non-limiting example, the context component 326 may identify any m of the plurality of documents 904 that have the most similar or closest embedding to the embedding 802, and such m documents may be considered the set of relevant documents 702 for any suitable positive integer m. That is, the set of relevant documents 702 may include m documents, i.e., related document 702(1) through related document 702(m). In other words, related document 702(1) may be any of multiple documents 904 whose embedding is closest or most similar to embedding 802, while related document 702(m) may be any of multiple documents 904 whose embedding is m-th closest or m-th most similar to embedding 802.

[0169] In either case, the set of related documents 702 can be considered to be substantially or semantically related in some way to the natural language instructions 308, the image 402, the energy spectrum 404, or the current state of health 406. As a non-limiting example, assume that the natural language instructions 308 request or command that the beam voltage and stage temperature of a charged particle microscope be set to certain desired values. In such a case, any of the set of related documents 702 can be structured or unstructured text that describes or explains: how the beam voltage setting or stage temperature setting of the charged particle microscope 302 can be manipulated or adjusted; how it is known or expected that manipulating or adjusting the beam voltage setting or stage temperature setting will affect the charged particle microscope 302 given the current state of health 406; or how it is known or expected that a sample whose captured image or energy spectrum matches or appears to match the image 402 or energy spectrum 404 will respond to a change in beam voltage or a change in stage temperature given the current state of health 406. Thus, the set of relevant documents 702 can be viewed as providing valuable contextual information about the natural language instructions 308 , the image 402 , the energy spectrum 404 , or the current health state 406 .

[0170] Consider now FIG. 10. In various embodiments, there may be a set of available deep learning models 1002. In various aspects, the set of available deep learning models 1002 may include p models, i.e., available deep learning models 1002(1) through available deep learning models 1002(p), for any suitable positive integer p. In various instances, any of the set of available deep learning models 1002 may exhibit any suitable deep learning internal architecture. Indeed, in various instances, any of the set of available deep learning models 1002 may have an input layer, one or more hidden layers, and an output layer. In various instances, any of such layers may be connected together by any suitable inter-neuron or inter-layer connections (e.g., forward connections, skip connections, or recurrent connections). Moreover, in various cases, any of such layers may be any suitable type of neural network layer having any suitable learnable or trainable internal parameters (e.g., any of such input layers, one or more hidden layers, or output layers may be convolutional layers, dense layers, batch normalization layers, LSTM layers, or transformer layers). Still further, in various cases, any of such layers may be any suitable type of neural network layer having any suitable fixed or non-trainable internal parameters (e.g., any of such input layers, one or more hidden layers, or output layers may be nonlinear layers, padding layers, pooling layers, or concatenation layers).

[0171] Regardless of their particular internal architecture, each of the set of available deep learning models 1002 may be configured to perform a respective inference task on an input sample image or on an input sample energy spectrum. In various aspects, the inference task may be any suitable computerized prediction or forecast of any suitable details about any sample whose image or energy spectrum is being analyzed. As a non-limiting example, any of the set of available deep learning models 1002 may be configured to perform classification on an input sample image or on an input sample energy spectrum to classify a given sample into one of two or more defined classes or categories (e.g., a chemical composition category, a failure mode category, a structure present category, a structure absent category). As another non-limiting example, any of the set of available deep learning models 1002 may be configured to perform segmentation on an input sample image or on an input energy spectrum to classify each individual portion (e.g., individual pixels / voxels or groups thereof, individual energy bins or groups thereof) of such input image or energy spectrum into one of two or more defined classes or categories. As yet another non-limiting example, any of the set of available deep learning models 1002 may be configured to perform regression on input sample images or on input energy spectra to calculate one or more continuously varying scalars, vectors, matrices, or tensors for each such input image or energy spectrum (e.g., regression may include resolution enhancement, noise removal, bounding box calculation, or any other suitable data transformation). It should be understood that different ones of the set of available deep learning models 1002 may be configured to perform different inference tasks on input images or input energy spectra.It should further be appreciated that any of the set of available deep learning models 1002 may be configured to receive only an image 402 as an input, any of the set of available deep learning models 1002 may be configured to receive only an energy spectrum 404 as an input, or any of the set of available deep learning models 1002 may be configured to receive both an image 402 and an energy spectrum 404 as input.

[0172] In various aspects, the context component 326 can electronically generate a set of inference task results 704 by running a set of available deep learning models 1002 on the image 402 or on the energy spectrum 404, respectively.

[0173] As a non-limiting example, the context component 326 can execute an available deep learning model 1002(1) on the image 402 or on the energy spectrum 404, and such execution can produce an inference task result 704(1). More specifically, the context component 326 can feed the image 402 or the energy spectrum 404 (either alone or a concatenation of both) into an input layer of the available deep learning model 1002(1), such input data can complete a forward pass through one or more hidden layers of the available deep learning model 1002(1), and the output layer of the available deep learning model 1002(1) can calculate or otherwise compute the inference task result 704(1) based on activation maps or feature maps provided by the one or more hidden layers of the available deep learning model 1002(1). Thus, the inference task result 704 can be any classification label, segmentation mask, or regression output that the available deep learning model 1002(1) predicts or infers for the image 402 or for the energy spectrum 404.

[0174] As another non-limiting example, the context component 326 can execute an available deep learning model 1002(p) on the image 402 or on the energy spectrum 404, and such execution can produce an inference task result 704(p). More specifically, the context component 326 can feed the image 402 or the energy spectrum 404 (either alone or a concatenation of both) into an input layer of the available deep learning model 1002(p), such input data can complete a forward pass through one or more hidden layers of the available deep learning model 1002(p), and the output layer of the available deep learning model 1002(p) can calculate or otherwise compute the inference task result 704(p) based on the activation maps or feature maps provided by the one or more hidden layers of the available deep learning model 1002(p). Thus, the inference task result 704 can be any classification label, segmentation mask, or regression output that the available deep learning model 1002(p) predicts or infers for the image 402 or for the energy spectrum 404.

[0175] In various cases, inference task results 704(1) through 704(p) can be collectively considered a set of inference task results 704.

[0176] The set of inference task results 704 can therefore be viewed as providing valuable complementary information about the image 402 or energy spectrum 404 and, therefore, about the currently loaded sample 306 .

[0177] Now consider FIG. 11. In various embodiments, there may be a digital twin prompt 1102. In various aspects, the digital twin prompt 1102 may be one or more unstructured or plain text sentences or sentence fragments that request or command the digital twin 602 to run or conduct a virtual experiment related to the natural language instructions 308. As a non-limiting example, for any suitable positive integer q, the digital twin prompt 1102 may be the following sentence: "The specimen is characterized by an image 402 or an energy spectrum 404. The microscope is characterized by its current health state 406. Design q different experiments to test how the specimen or microscope would react if the digital twin 602 were to follow the natural language instructions 308."

[0178] As such, in various instances, the context component 326 can electronically execute the LLM 310 against the natural language instructions 308, against the image 402, against the energy spectrum 404, against the current health state 406, against the digital twin prompt 1102, or against any suitable combination thereof. In various cases, such execution can cause the LLM 310 to generate a set of digital twin function calls 1104. More specifically, the context component 326 can concatenate together the natural language instructions 308, the image 402, the energy spectrum 404, the current health state 406, and the digital twin prompt 1102. Note that in some cases, the concatenation can include any other combination of the foregoing (e.g., can include less than all of the natural language instructions 308, the image 402, the energy spectrum 404, the current health state 406, and the digital twin prompt 1102). In various aspects, the context component 326 can feed the concatenation to an input layer of the encoder portion 312. In various aspects, the concatenation may complete a forward pass through one or more hidden layers of the encoder portion 312. In various cases, the output layer of the encoder portion 312 may compute or otherwise calculate one or more embeddings (not shown) based on activation or feature maps provided by the one or more hidden layers of the encoder portion 312. In various cases, those one or more embeddings may be routed to the input layer of the synthesizer portion 314. In various aspects, those one or more embeddings may complete a forward pass through one or more hidden layers of the synthesizer portion 314, and the output layer of the synthesizer portion 314 may compute or otherwise calculate a set of digital twin function calls 1104 based on the activation or feature maps provided by the one or more hidden layers of the synthesizer portion 314.

[0179] In various instances, as described above, digital twin prompt 1102 may request or command the creation of q distinct virtual experiments to be performed by digital twin 602. Accordingly, set of digital twin function calls 1104 may include q calls, i.e., digital twin function call 1104(1) through digital twin function call 1104(q). In various instances, each of the set of digital twin function calls 1104 may be considered to be one or more respective lines of composed computer code that define a respective virtual experiment (e.g., in any syntax understood by the digital twin, such as JSON code or Python code) that is readable by and executable by digital twin 602. As a non-limiting example, digital twin function call 1104(1) may be one or more first lines of composed computer code that define a first virtual experiment that may be run on digital twin 602. In other words, digital twin function call 1104(1) can specify that the first value the LLM 310 infers to parse the natural language instruction 308 should be assigned to a set of input variables 606 or to a parametric state 604. As another non-limiting example, digital twin function call 1104(q) can be one or more qth lines of synthesized computer code that define a qth virtual experiment that may be run on the digital twin 602. That is, digital twin function call 1104(q) can specify that the qth value the LLM 310 infers to parse the natural language instruction 308 should be assumed or assigned by a set of input variables 606 or by a parametric state 604.

[0180] 12. In various embodiments, the context component 326 can electronically command, direct, or otherwise cause the digital twin 602 to run, perform, or execute each of the set of digital twin function calls 1104, thereby resulting in a set of simulation results 706.

[0181] As a non-limiting example, the context component 326 can cause the digital twin 602 to execute the digital twin function call 1104(1). That is, the context component 326 can cause the digital twin 602 to assign any values ​​specified in the digital twin function call 1104(1) to the set of input variables 606 or the set of parametric states 604, so that the digital twin 602 can run a first virtual experiment using those newly assigned values. In various aspects, such a first virtual experiment can produce simulation results 706(1), which can be either the set of output variables 608 that the digital twin 602 computed for the digital twin function call 1104(1) or because of the digital twin function call 1104(1). Note that in some cases, simulation results 706(1) can be considered to be indicative of what the health state of charged particle microscope 302 would be if natural language instructions 308 were to be performed according to whatever first usage scenario is defined or specified by digital twin function call 1104(1).

[0182] As another non-limiting example, the context component 326 can cause the digital twin 602 to execute a digital twin function call 1104(q). That is, the context component 326 can cause the digital twin 602 to assign any values ​​specified in all of the digital twin functions 1104(q) to the set of input variables 606 or the set of parametric states 604, so that the digital twin 602 can run the qth virtual experiment using those newly assigned values. In various aspects, such a qth virtual experiment can produce a simulation result 706(q), which can be any set of output variables 608 that the digital twin 602 computed for or because of the digital twin function call 1104(q). As noted above, in some cases, the simulation results 706(q) can be considered to indicate what the health state of the charged particle microscope 302 would be if the natural language instructions 308 were to be performed according to any qth usage scenario defined or specified by the digital twin function call 1104(q).

[0183] In various cases, simulation results 706(1) through 706(q) can be collectively considered as a set of simulation results 706.

[0184] Therefore, the set of simulation results 706 can be viewed as providing valuable supplemental information regarding the charged particle microscope 302, natural language instructions 308, images 402, or energy spectra 404.

[0185] FIG. 13 illustrates a block diagram of an exemplary, non-limiting system including natural language responses and synthesized code that can facilitate large-scale language model support for charged particle microscope operations in accordance with one or more embodiments described herein.

[0186] In various embodiments, the model component 328 can utilize the LLM 310 to electronically generate a natural language response 1302 or synthesized code 1304 based on the natural language instruction 308 and based on either the information or data collected or obtained by the state component 324 or by the context component 326. Various non-limiting aspects are described with respect to FIG.

[0187] As shown in FIG. 14 , there may be a damage prompt 1401. In various aspects, the damage prompt 1401 may be one or more unstructured or plain text sentences or sentence fragments that request or command that the natural language instructions 308 be evaluated for the likelihood of specimen damage. In other words, the damage prompt 1401 may ask whether fulfilling the natural language instructions 308 will cause damage, or whether it is likely to cause damage, to the currently loaded specimen 306. As a non-limiting example, the damage prompt 1401 may be the following sentence: "The specimen is characterized by the image 402 or the energy spectrum 404. The microscope is characterized by the current health state 406. Will executing the natural language instructions 308 on the microscope damage the specimen?" As another non-limiting example, the damage prompt 1401 may be the following sentence: "Determine whether following the natural language instructions 308 will harm the specimen shown in the image 402 or the energy spectrum 404."

[0188] Thus, in various instances, the model component 328 can electronically execute the LLM 310 against the natural language instructions 308, against the image 402, against the energy spectrum 404, against the current health state 406, against the set of related documents 702, against the set of inference task results 704, against the set of simulation results 706, against the damage prompt 1401, or any suitable combination thereof. In various cases, such execution can cause the LLM 310 to generate a natural language response 1302 or a synthesized code 1304. More specifically, the context component 326 can concatenate the natural language instructions 308, the image 402, the energy spectrum 404, the current health state 406, the set of related documents 702, the set of inference task results 704, the set of simulation results 706, and the damage prompt 1401 together. As noted above, in various cases, the concatenation may include any other combination of the foregoing (e.g., may include less than all of the natural language instructions 308, the image 402, the energy spectrum 404, the current health state 406, the set of related documents 702, the set of inference task results 704, the set of simulation results 706, and the damage prompt 1401). In various cases, the model component 328 may feed the concatenation to an input layer of the encoder portion 312. In various aspects, the concatenation may complete a forward pass through one or more hidden layers of the encoder portion 312. In various cases, the output layer of the encoder portion 312 may compute or otherwise calculate one or more embeddings (not shown) based on activation or feature maps provided by the one or more hidden layers of the encoder portion 312. In various cases, the one or more embeddings may be routed to an input layer of the synthesizer portion 314.In various aspects, the one or more embeddings can complete a forward pass through one or more hidden layers of the synthesizer portion 314, and the output layer of the synthesizer portion 314 can compute or otherwise calculate a natural language response 1302 or synthesized code 1304 based on the activation or feature maps provided by the one or more hidden layers of the synthesizer portion 314.

[0189] In various aspects, as shown, the natural language response 1302 can include a specimen damage explanation 1402. In various instances, the specimen damage explanation 1402 can be one or more unstructured or plain text declarative sentences or sentence fragments that semantically answer the damage prompt 1401. That is, the specimen damage explanation 1402 can be synthesized text that describes or states whether following, fulfilling, or otherwise implementing the natural language command 308 (as requested by a user or technician of the charged particle microscope 302) will degrade, harm, damage, or otherwise damage the currently loaded specimen 306.

[0190] As a non-limiting example, assume that the natural language instruction 308 requests or commands that the beam voltage of the charged particle microscope 302 be set to 50 kV, and assume that the currently loaded sample 306 (if unknown to the user or technician) has a chemical composition that will be adversely affected or harmed by exposure to the 50 kV beam. As described above, the image 402 or energy spectrum 404 can be considered to convey at least some information regarding the chemical, physical, or compositional properties of the currently loaded sample 306. Because the LLM 310 can be trained on (e.g., receive as input) the image 402 or energy spectrum 404, the LLM 310 can be considered to have access to that chemical, physical, or compositional information. Thus, the LLM 310 can infer or predict that the currently loaded sample 306 is likely to be harmed by the natural language instruction 308, and therefore, the LLM 310 can synthesize a sample damage description 1402 to state or describe that such harm will occur if the natural language instruction 308 is followed. It should be noted that the set of related documents 702, the set of inference task results 704, or the set of simulation results 706 can be considered as additional, supplemental, contextual, or otherwise enriching information that assists or assists the LLM 310 in correctly or accurately synthesizing the specimen damage description 1402 (e.g., correctly or accurately inferring or predicting that the currently loaded specimen 306 will be damaged by the natural language instruction 308).

[0191] As another non-limiting example, assume that the natural language instruction 308 requests or commands that the stage temperature of the charged particle microscope 302 be set to 400 Kelvin, and assume that the currently loaded sample 306 (if unknown to the user or technician) has a chemical composition that will be adversely affected or harmed by exposure to such high temperatures. Again, the image 402 or energy spectrum 404 can be considered to convey at least some information regarding the chemical, physical, or compositional properties of the currently loaded sample 306. Because the LLM 310 can be conditioned on the image 402 or energy spectrum 404, the LLM 310 can thus infer or predict that the currently loaded sample 306 is likely to be harmed by the natural language instruction 308, and can accordingly synthesize a sample damage description 1402 to state or describe that harm will occur. As noted above, the set of related documents 702, the set of inference task results 704, or the set of simulation results 706 can be considered additional, supplemental, contextual, or otherwise enriching information that aids or assists the LLM 310 in correctly or accurately synthesizing the specimen damage description 1402.

[0192] Here, in some cases, the damage prompt 1401 may ask not only whether the natural language instruction 308 will harm the currently loaded specimen 306, but also whether the natural language instruction 308 will harm the charged particle microscope 302 itself. As a non-limiting example, the damage prompt 1401 may include the following sentence: "The specimen is characterized by an image 402 or an energy spectrum 404. The microscope is characterized by a current health state 406. Will the natural language instruction 308 wear or damage the specimen or microscope?" In such a situation, the natural language response 1302 may include a microscope damage description 1404. In various instances, the microscope damage description 1404 may be one or more unstructured or plain text declarative sentences or sentence fragments that semantically answer the damage prompt 1401 regarding the charged particle microscope 302. That is, the microscope damage description 1404 may be synthesized text that describes or states whether following, fulfilling, or otherwise implementing the natural language command 308 (as requested by a user or technician) will degrade, harm, wear, or otherwise damage the charged particle microscope 302 itself.

[0193] Consider again the non-limiting example above in which the natural language command 308 requests or commands that the beam voltage of the charged particle microscope 302 be set to 50 kV. Now, assume (if unknown to the user or technician) that the charged particle microscope 302 has a nearly worn cathode ray tube, and that attempting 50 kV with a nearly worn cathode ray tube may expose the charged particle microscope 302 to excessive wear or damage. In various cases, the current health state 406 can be considered to convey (directly or indirectly) the nearly worn cathode ray tube of the charged particle microscope 302. Because the LLM 310 can be adjusted with the current health state 406, the LLM 310 can be considered to have access to that portion of information regarding the nearly worn cathode ray tube. Thus, the LLM 310 can infer or predict that the charged particle microscope 302 is likely to be harmed or damaged by the natural language instructions 308, and therefore the LLM 310 can synthesize a microscope damage description 1404 that states or describes that such harm will occur. Note that the set of related documents 702 or the set of simulation results 706 can be considered additional, supplemental, contextual, or otherwise enriching information that aids or assists the LLM 310 in correctly or accurately synthesizing the specimen damage description 1404 (e.g., correctly or accurately inferring or predicting that the charged particle microscope 302 will be harmed by the natural language instructions 308).

[0194] As another non-limiting example, assume that the natural language instruction 308 requests or commands that the vacuum chamber of the charged particle microscope 302 be set to 0.0001 Pascals, and assume that the currently loaded sample 306 (if unknown to the user or technician) has a chemical or physical composition that would rupture at such pressure, potentially damaging the charged particle microscope 302. As discussed above, the image 402 or energy spectrum 404 can be considered to convey at least some information regarding the chemical, physical, or compositional properties of the currently loaded sample 306. Because the LLM 310 can be tuned with the image 402 or energy spectrum 404, the LLM 310 can be considered to have access to that chemical, physical, or compositional information and can therefore infer or predict that the currently loaded sample 306 is likely to rupture due to the natural language instruction 308, thereby damaging the charged particle microscope 302, and can therefore synthesize a microscope damage description 1404 to state or describe that such damage may occur. Again, the set of related documents 702, the set of inference task results 704, or the set of simulation results 706 can be considered as additional, supplemental, contextual, or otherwise enriching information that aids or assists the LLM 310 in correctly or accurately synthesizing the specimen damage description 1404.

[0195] In various aspects, as described above, the natural language instruction 308 can request or command that one or more of the plurality of configurable operational settings 304 be set, changed, or adjusted to one or more target values ​​or states 332. In situations where the LLM 310 infers or predicts that following the natural language instruction 308 will harm the currently loaded sample 306 or harm the charged particle microscope 302, the natural language response 1302 can, in various aspects, include one or more recommended values ​​or states 1406. In various instances, the one or more recommended values ​​or states 1406 can be considered as alternative versions of the one or more target values ​​or states 332 that the LLM 310 infers or predicts will not harm or damage the currently loaded sample 306 or the charged particle microscope 302.

[0196] As a non-limiting example, again assume that the natural language instruction 308 requests or commands a beam voltage of 50 kV. Further assume that the LLM 310 infers or predicts (based on the image 402 or energy spectrum 404 supplemented by the current health state 406, the set of related documents 702, the set of inference task results 704, the set of simulation results 706, or any combination thereof) that a beam voltage of 50 kV will harm either the currently loaded sample 306 or the charged particle microscope 302. In such a case, the LLM 310 can infer or predict (based on 402 or 404 supplemented by 406, 702, 704, 706, or any combination thereof) that such damage can be avoided by a beam current of 15 kV (e.g., here, 15 kV can be considered one or more recommended values ​​or states 1406). Thus, the LLM 310 can synthesize a natural language response 1302 to state, describe, or explain that a 50 kV beam voltage can be avoided by using a 15 kV beam voltage instead if any sample or microscope damage would be caused.

[0197] In some embodiments, as shown, the natural language response 1302 can include a skipped step description 1408. In various instances, the skipped step description 1408 can be one or more unstructured or plain text sentences or sentence fragments that describe, state, or otherwise identify one or more steps, actions, or tasks that a user or technician can perform with respect to the charged particle microscope 302 or with respect to the currently loaded sample 306 to avoid any damage that is inferred or predicted to be caused by the natural language instruction 308. That is, the LLM 310 can infer or predict, in some cases, that the natural language instruction 308 will not cause damage to the currently loaded sample 306 or to the charged particle microscope 302 only after performance of some other step, action, or task that the user or technician has not yet performed (e.g., that the user or technician inadvertently skipped or forgot).

[0198] As a non-limiting example, assume that the natural language instruction 308 requests or commands the charged particle microscope 302 to perform a voltage contrast analysis, and assume (if unknown to the user or technician) that the currently loaded sample 306 should first be cleaned (e.g., with an alcohol solution) prior to the voltage contrast analysis. As described above, the image 402 or energy spectrum 404 can be considered to convey at least some information regarding the chemical, physical, or compositional properties of the currently loaded sample 306. Because the LLM 310 can be adjusted with the image 402 or energy spectrum 404, the LLM 310 can thus infer or predict that the currently loaded sample 306 is contaminated or not yet clean and that cleaning is required prior to the voltage contrast analysis. Accordingly, the LLM 310 can have the skipped step explanation 1408 textually state or describe such an inference. Again, the set of related documents 702, the set of inference task results 704, or the set of simulation results 706 can be considered additional, supplemental, contextual, or otherwise enriching information that assists or assists the LLM 310 in correctly or accurately synthesizing the skipped step explanation 1408 (e.g., correctly or accurately inferring or predicting that the currently loaded sample 306 is not yet clean and that such cleaning is required before voltage contrast analysis can begin).

[0199] In various aspects, as described above, the LLM 310 can generate synthesized code 1304 in addition to (or in some cases instead of) the natural language response 1302. In various instances, the synthesized code 1304 can be one or more lines of computer code (e.g., Python code, C++ code, JSON code) that can perform (when executed, run, or compiled) any suitable computerized action with respect to the charged particle microscope 302 or with respect to the natural language response 1302. As a non-limiting example, in a situation where the natural language response 1302 includes one or more recommended values ​​or states 1406, execution of the synthesized code 1304 can cause the charged particle microscope 302 to automatically set, change, or adjust one or more of the plurality of configurable operational settings 304 to the one or more recommended values ​​or states 1406 instead of one or more target values ​​or states 332.

[0200] In various embodiments, the presenter component 330 can electronically perform any suitable action with respect to the natural language response 1302 or with respect to the synthesized code 1304. As a non-limiting example, the presenter component 330 can visually render the natural language response 1302 on any suitable electronic display (e.g., a computer screen) of the charged particle microscope 302 or of any suitable computerized workstation associated with the charged particle microscope 302. In such a case, the natural language response 1302 can be visible to or viewed by a user or technician of the charged particle microscope 302. As another non-limiting example, the presenter component 330 can audibly play the natural language response 1302 on any suitable electronic speaker of the charged particle microscope 302 or of any suitable computerized workstation associated with the charged particle microscope 302. In such a case, the natural language response 1302 can be heard by or heard by a user or technician of the charged particle microscope 302. In either case, the presenter component 330 can be thought of as warning or informing the user or technician of unintended or unwanted harm that may occur if the natural language command is fulfilled or if the natural language command is attempted to be followed. Thus, even a user or technician who is inexperienced or untrained in the field of charged particle microscopy can nevertheless operate the charged particle microscope 302 without much fear of unintentionally damaging the currently loaded sample 306 or the charged particle microscope 302 itself. In this way, the system 316 can be thought of as helping to protect the currently loaded sample 306 or the charged particle microscope 302 itself from the inexperience of the user or technician.

[0201] In some instances, the presenter component 330 can electronically execute, run, or compile the synthesized code 1304. In various cases, this can cause the charged particle microscope 302 to automatically implement one or more recommended values ​​or states 1406 instead of one or more target values ​​or states 332. In other words, the microscopy settings requested by the user or technician can be ignored, and the microscopy settings inferred by the LLM 310 for the currently loaded sample 306 can be automatically implemented. Again, the system 316 can be viewed as helping to protect the currently loaded sample 306 or the charged particle microscope 302 itself from the inexperience of the user or technician.

[0202] In some embodiments, the damage prompt 1401 may be replaced with a more general result prompt. That is, rather than specifically asking whether the implementation of the natural language instruction 308 will harm the currently loaded sample 306 or the charged particle microscope 302, the result prompt may instead ask how the currently loaded sample 306 or the charged particle microscope 302 will respond to the implementation of the natural language instruction 308. In other words, the result prompt may be one or more unstructured or plain text sentences or sentence fragments that request or command the identification of any results (good or bad, positive or negative) that are likely to occur with respect to the currently loaded sample 306 or with respect to the charged particle microscope 302 if the natural language instruction 308 were to be followed. As a non-limiting example, the result prompt may be the following sentence: "The sample is characterized by an image 402 or energy spectrum 404. The microscope is characterized by a current health state 406. How will the implementation of the natural language instruction 308 affect the sample or microscope?" As another non-limiting example, the result prompt may be the following statement: "The sample is characterized by an image 402 or an energy spectrum 404. The microscope is characterized by its current health state 406. Tell me what will happen to the sample or microscope if I follow the natural language command 308."

[0203] In such a case, the specimen damage description 1402 may be replaced with a more general specimen result description, and the microscope damage description 1404 may similarly be replaced with a more general microscope result description. In various aspects, the specimen result description may be one or more synthesized declarative sentences or sentence fragments that describe or state how the currently loaded specimen 306 would be affected if the natural language instructions 308 were followed, fulfilled, or otherwise implemented. Similarly, the microscope result description may be one or more synthesized declarative sentences or sentence fragments that describe or state how the charged particle microscope 302 would be affected if the natural language instructions 308 were followed, fulfilled, or otherwise implemented.

[0204] As a non-limiting example, assume that the natural language command 308 requests or commands that the beam voltage of the charged particle microscope 302 be set to 50 kV, and assume that the currently loaded sample 306 has a chemical composition (unknown to the user or technician) that will not be damaged or degraded by exposure to the 50 kV beam, but will be unevenly charged by exposure to the 50 kV beam (thereby leading to undesirable charging artifacts). As described above, the image 402 or energy spectrum 404 can be considered to convey at least some information regarding the chemical, physical, or compositional properties of the currently loaded sample 306. Because the LLM 310 can be conditioned on (e.g., receive as input) the image 402 or energy spectrum 404, the LLM 310 can be considered to have access to that chemical, physical, or compositional information. Thus, the LLM 310 can infer or predict that the currently loaded specimen 306 is likely to be non-uniformly charged, and therefore the LLM 310 can synthesize a specimen result explanation to state or describe that such charging will occur when the natural language instructions 308 are followed. As noted above, the set of related documents 702, the set of inference task results 704, or the set of simulation results 706 can be considered additional, supplemental, contextual, or otherwise enriching information that aids or assists the LLM 310 in correctly or accurately synthesizing a specimen damage explanation (e.g., correctly or accurately inferring or predicting that the currently loaded specimen 306 will be non-uniformly charged per the natural language instructions 308).

[0205] In this manner, the various embodiments described herein can be viewed as predicting in advance the likely outcome, if not damage or wear, that the execution of the natural language command 308 will have on the currently loaded sample 306 or on the charged particle microscope 302. Again, the system 316 can be viewed as helping to protect the user or technician from their own microscopy inexperience in real time.

[0206] 15 and 16 illustrate an exemplary, non-limiting flow diagram of a computer-implemented method according to one or more embodiments described herein.

[0207] Consider first Figure 15. In various embodiments, operation 1502 may include accessing (e.g., via 322) by a device operably coupled to a processor (e.g., 318) a natural language request (e.g., 308) to set a controllable operating parameter (e.g., one of 304) of a charged particle microscope (e.g., 302) to a desired value (e.g., one of 332).

[0208] In various aspects, operation 1504 can include causing the device (e.g., via 324) to respond to a natural language request to cause the charged particle microscope to capture an image (e.g., 402) or energy spectrum (e.g., 404) of a currently loaded sample (e.g., 306) using a default microscopy protocol.

[0209] In various instances, operation 1506 may include accessing, by the device (e.g., via 324) in response to a natural language request, the current health state (e.g., 406) of the charged particle microscope from a digital twin (e.g., 602) that is synchronized with the charged particle microscope.

[0210] In various cases, operation 1508 may include identifying, by the device (e.g., via 326) and via an embedded search of a document database (e.g., 902), one or more documents (e.g., 702) related to the natural language request, to the image or energy spectrum, or to the current health condition.

[0211] In various aspects, operation 1510 may include executing, by the device (e.g., via 326), one or more available deep learning models (e.g., 1002) on the image or energy spectrum, thereby producing one or more inference task results (e.g., 704). In various cases, as shown, computer-implemented method 1500 may proceed to operation 1602 of computer-implemented method 1600.

[0212] Now consider Figure 16. In various embodiments, operation 1602 can include operating the digital twin by the device (e.g., via 326) according to one or more function calls (e.g., 1104) generated by the large-scale language model (e.g., 310) based on the natural language request, on the image or energy spectrum, or on the current health state. In various cases, this can produce one or more simulation results (e.g., 706) in the digital twin.

[0213] In various aspects, operation 1604 may include running, by the device (e.g., via 328), a large-scale language model against the natural language request, against the image or energy spectrum, against the current health state, against one or more documents, against one or more inference task results, or against one or more simulation results. In various cases, this may cause the large-scale language model to produce a natural language response (e.g., 1302, including 1402) indicating whether setting a controllable operational parameter to a desired value will harm the currently loaded sample.

[0214] In various instances, operation 1606 may include the device (e.g., via 330) visually or audibly rendering a natural language response on an electronic display or electronic speaker associated with the charged particle microscope. In this manner, an untrained or inexperienced user of the charged particle microscope may be alerted or notified in real time whether their request will unknowingly or unintentionally damage a currently loaded specimen.

[0215] Thus far, various embodiments have been described with respect to natural language instructions 308, i.e., various embodiments in which a user or technician requests or commands the charged particle microscope 302 to adjust settings of the charged particle microscope 302 in a particle manner or otherwise perform some particular microscopy task. However, a user or technician may have various other types of requests for the charged particle microscope 302. Non-limiting examples of such various other types of requests are described with respect to FIGS. 17-30.

[0216] Consider first Figure 17. Rather than providing natural language instructions 308, a user or technician of the charged particle microscope 302 can instead provide a natural language workflow query 1702.

[0217] In various aspects, the natural language workflow query 1702 may be any suitable number of plain text or unstructured sentences or sentence fragments that request or instruct identification of how to properly perform, conduct, or otherwise implement any suitable microscopy workflow on or using the charged particle microscope 302. As some non-limiting examples, the natural language workflow query 1702 may be any of the following: "Describe how to perform a voltage contrast analysis on this microscope," "What steps are required for a bend analysis?", or "How should a diffraction analysis be performed?"

[0218] As described above, the natural language workflow query 1702 may be provided or entered by a user or technician of the charged particle microscope 302 via any suitable human-computer interface device associated with the charged particle microscope 302 (e.g., the natural language workflow query 1702 may be typed or spoken by the user or technician).

[0219] Also as described above, the status component 324 may electronically acquire, receive, retrieve, or otherwise access an image 402, an energy spectrum 404, or a current health state 406 in response to receiving or accessing a natural language workflow query 1702.

[0220] Furthermore, as described above, the context component 326 can electronically acquire, receive, retrieve, or otherwise access a set of inference task results 704 by running a set of available deep learning models 1002 on the image 402 or on the energy spectrum 404, respectively.

[0221] Here, in various cases, the context component 326 can electronically identify the set of relevant documents 702 based on the natural language workflow query 1702 instead of based on the natural language instructions 308. As a non-limiting example, the context component 326 can concatenate together the natural language workflow query 1702, the image 402, the energy spectrum 404, the current health state 406, or any combination thereof, and execute the LLM 310 on the concatenation. Such execution can cause the LLM 310 to generate a particular embedding (e.g., as described with respect to FIG. 8 ) that substantially or semantically represents the natural language workflow query 1702, the image 402, the energy spectrum 404, or the current health state 406. Accordingly, the context component 326 can perform an embedding search in the document repository 902 to identify the set of relevant documents 702. In such cases, the set of relevant documents 702 can be considered to be substantially or semantically related or relevant to the natural language workflow query 1702 instead of the natural language instructions 308. In other words, the set of relevant documents 702 is obtained as described with respect to FIGS. 8 and 9, but the natural language instructions 308 may be replaced by natural language workflow queries 1702.

[0222] Similarly, in various cases, the context component 326 can electronically obtain the set of simulation results 706 based on a natural language workflow query 1702 instead of based on a natural language instruction 308. As a non-limiting example, the digital twin prompt 1102 can be one or more unstructured or plain text sentences or sentence fragments that request or command that a virtual experiment related to the natural language workflow query 1702 be run or conducted by the digital twin 602. For example, the digital twin prompt 1102 can be the following statement: "The specimen is characterized by an image 402 or an energy spectrum 404. The microscope is characterized by a current health state 406. Design q different experiments for the digital twin 602 to test how the specimen or microscope responds to the microscopy workflows identified in the natural language workflow query 1702." Thus, the context component 326 can concatenate together the natural language workflow query 1702, the image 402, the energy spectrum 404, the current health state 406, the digital twin prompt 1102, or any combination thereof, and execute the LLM 310 on the concatenation. Such execution can cause the LLM 310 to generate a set of digital twin function calls (e.g., as described with respect to FIG. 11 ) related to the natural language workflow query 1702, the image 402, the energy spectrum 404, or the current health state 406. Thus, the context component 326 can cause the digital twin 602 to execute its set of function calls, thereby producing a set of simulation results 706. In such a case, the set of simulation results 706 can be viewed as resulting from a virtual experiment testing or probing whatever microscope workflow is specified in the natural language workflow query 1702, instead of resulting from a virtual experiment testing or probing the natural language instructions 308. In other words, the set of simulation results 706 is obtained as described with respect to Figures 11 and 12, but the natural language instructions 308 can be replaced by the natural language workflow query 1702.

[0223] Thus, in various aspects, the model component 328 can electronically execute the LLM 310 against a concatenation of a natural language workflow query 1702, an image 402, an energy spectrum 404, a current health state 406, a set of related documents 702, a set of inference task results 704, or a set of simulation results 706, such execution causing the LLM 310 to generate a natural language response 1302 or synthesized code 1304.

[0224] In such embodiments, the natural language response 1302 can include a workflow tutorial 1704. In various aspects, the workflow tutorial 1704 can be one or more unstructured or plain text declarative sentences or sentence fragments that semantically answer the natural language workflow query 1702. That is, the workflow tutorial 1704 can be synthesized text that describes, states, or lists (e.g., as bullet points) what sequence of steps is or is required to perform the particular microscopy workflow specified in the natural language workflow query 1702. In various instances, some microscopy workflows can be sample-dependent. Thus, the workflow tutorial 1704 can be thought of as describing or instructing which particular sequence steps are or are required to properly perform the specified microscopy workflow for the currently loaded sample 306, even though the natural language workflow query 1702 may be completely silent with respect to the currently loaded sample 306.

[0225] As a non-limiting example, assume that the natural language workflow query 1702 asks about a voltage contrast analysis. Further assume that the voltage contrast analysis may require steps A, B, and C to be performed sequentially for an organic sample, while the voltage contrast analysis may instead require steps D, B, E, and C to be performed sequentially for an inorganic sample. As discussed above, the image 402 or energy spectrum 404 can be considered to convey at least some information about the chemical, physical, or compositional properties of the currently loaded sample 306. Because the LLM 310 can be tuned on the image 402 or energy spectrum 404, the LLM 310 can be considered to have access to that chemical, physical, or compositional information. Thus, the LLM 310 can infer or predict whether the currently loaded sample 306 is organic or inorganic, and accordingly, the LLM 310 can synthesize a workflow tutorial 1704 to present, describe, or enumerate the appropriate sequence of voltage contrast analysis steps (e.g., if the LLM 310 infers that the currently loaded sample 306 is organic, the workflow tutorial 1704 can teach the sequence of steps A, B, and C; if, instead, the LLM 310 infers that the currently loaded sample 306 is inorganic, the workflow tutorial 1704 can instead teach the sequence of steps D, B, E, and C). As noted above, it should be noted that the set of related documents 702, the set of inference task results 704, or the set of simulation results 706 can be considered additional, supplemental, contextual, or otherwise enriching information that assists or assists the LLM 310 in correctly or accurately synthesizing the workflow tutorial 1704 (e.g., correctly or accurately inferring or predicting what sequence of steps is or will be required to properly perform a specified microscopy workflow for the currently loaded sample 306).

[0226] In some cases, it may be possible that the workflow specified in the natural language workflow query 1702 may be inappropriate or otherwise not operable for the currently loaded sample 306 (e.g., due to a user's or technician's lack of training, skill, or experience) (e.g., some workflows may not be possible for samples containing particular chemicals or exhibiting particular physical structures). In such cases, the LLM 310 may infer or predict (due to the image 402, the energy spectrum 404, or any of their supplementary or contextual data) that the specified microscopy workflow cannot be properly performed on the currently loaded sample 306 at all, and the workflow tutorial 1704 may state or describe accordingly. In various aspects, the LLM 310 may infer or predict (again, due to the image 402, the energy spectrum 404, or any of their supplementary or contextual data) some alternative microscopy workflow that is operable for the currently loaded sample 306 or that is otherwise appropriate for the currently loaded sample 306. In such a case, the workflow tutorial 1704 may describe or explain any sequence of steps that are necessary or required to properly perform that alternative microscopy workflow on the currently loaded sample 306.

[0227] In various aspects, as described above, the LLM 310 can generate synthesized code 1304 in addition to (or in some cases instead of) the natural language response 1302. Also, as described above, the synthesized code 1304 can be one or more lines of computer code that can (when executed, run, or compiled) perform any suitable computerized action with respect to the charged particle microscope 302 or with respect to the natural language response 1302. As a non-limiting example, in a situation where the natural language response 1302 includes a workflow tutorial 1704, execution of the synthesized code 1304 can cause an electronic display of the charged particle microscope 302 (or any suitable computerized workstation associated with the charged particle microscope 302) to play one or more pre-recorded videographic visualizations or animations of the workflow tutorial 1704. For example, assume that the workflow tutorial 1704 textually describes or instructs how to properly perform voltage contrast analysis on the currently loaded sample 306, and assume that a database or storage bank of pre-recorded videos or animations for various microscopy workflows exists. In such a case, the LLM 310 can be thought of as identifying which particular video or animation in its database or storage bank corresponds to any particular step of the voltage contrast analysis that has been inferred to be appropriate or necessary for the currently loaded sample 306. The LLM 310 can then cause the composed code 1304 to reference or otherwise invoke those identified videos or animations, such that execution of the composed code 1304 can cause those identified videos or animations to be visually played on one or more computer screens (e.g., each video or animation can visually illustrate how one or more respective steps of the voltage contrast analysis are performed).

[0228] As described above, the presenter component 330 can electronically perform any suitable action with respect to the natural language response 1302 or with respect to the synthesized code 1304, such as visually rendering or audibly playing the natural language response 1302. In some cases, the presenter component 330 can electronically execute, run, or compile the synthesized code 1304, thereby playing any pre-recorded video or animation corresponding to the workflow tutorial 1704. Thus, even if a user or technician is inexperienced or untrained in the field of charged particle microscopy, the user or technician can nevertheless competently perform a microscopy workflow on or using the charged particle microscope 302 in a manner or style appropriate for the currently loaded sample 306. In this manner, the system 316 can be thought of as providing real-time, sample-tailored workflow guidance to the user or technician.

[0229] 18 and 19 illustrate an exemplary, non-limiting flow diagram of a computer-implemented method according to one or more embodiments described herein.

[0230] Consider first Figure 18. In various embodiments, operation 1802 may include accessing (e.g., via 322) by a device operably coupled to a processor (e.g., 318) a natural language request (e.g., 1702) asking how to perform a given microscopy workflow using a charged particle microscope (e.g., 302).

[0231] In various aspects, as shown, the computer-implemented method 1800 may proceed to operations 1504, 1506, 1508, and 1510 as described above. The computer-implemented method 1800 may then proceed to operation 1602 of the computer-implemented method 1900.

[0232] 19. In various cases, operation 1602 may be as described above, and the computer-implemented method 1900 may proceed to operation 1902. In various cases, operation 1902 may include running, by the device (e.g., via 328), a large-scale language model (e.g., 310) against the natural language request (e.g., 1702), against the image (e.g., 402) or energy spectrum (e.g., 404), against the current health state (e.g., 406), against one or more documents (e.g., 702), against one or more inference task results (e.g., 704), or against one or more simulation results (e.g., 706). In various cases, this may cause the large-scale language model to produce a natural language response (e.g., 1302, including 1704) that explains, describes, or instructs how to properly perform a given microscopy workflow for the currently loaded sample (e.g., 306).

[0233] As shown, the computer-implemented method 1900 may proceed to operation 1606 as described above.

[0234] 20. Rather than providing natural language instructions 308, a user or technician of the charged particle microscope 302 can instead provide a natural language malfunction query 2002.

[0235] In various aspects, the natural language malfunction query 2002 may be any suitable number of plain text or unstructured sentences or sentence fragments that request or command the identification of why the charged particle microscope 302 is experiencing any suitable microscopy malfunction symptom. In some cases, the natural language malfunction query 2002 may include a sample description 2004, which may be one or more unstructured or plain text sentences or sentence fragments that identify at least some known aspects, details, characteristics, or properties of the currently loaded sample 306. As some non-limiting examples, the natural language malfunction query 2002 may be any of the following: "I have a steel sample. Why is my image blurry?", "I have an organic sample. Why is the microscope raising error code ABC?", or "Tell me why the microscope is beeping on my printed circuit board sample."

[0236] As noted above, the natural language malfunction query 2002 may be provided or entered by a user or technician via any suitable human computer interface device associated with the charged particle microscope 302 .

[0237] Also as described above, the status component 324 may electronically acquire, receive, retrieve, or otherwise access an image 402, an energy spectrum 404, or a current health state 406 in response to receiving or accessing a natural language malfunction query 2002.

[0238] Furthermore, as described above, the context component 326 can electronically acquire, receive, retrieve, or otherwise access a set of inference task results 704 by running a set of available deep learning models 1002 on the image 402 or on the energy spectrum 404, respectively.

[0239] In various aspects, the context component 326 electronically retrieves, receives, searches, or otherwise accesses the set of relevant documents 702 as described with respect to FIGS. 8 and 9 , but the natural language instructions 308 may be replaced by natural language malfunction queries 2002.

[0240] Similarly, in various cases, the context component 326 electronically acquires, receives, retrieves, or otherwise accesses the set of simulation results 706, as described with respect to Figures 11 and 12, but the natural language instructions 308 may be replaced by a natural language malfunction query 2002. In some such cases, the digital twin prompt 1102 may be the following statement: "The specimen is characterized by an image 402 or energy spectrum 404. The microscope is characterized by a current health state 406. Design q different experiments for the digital twin 602 to probe the malfunction symptoms specified in the natural language malfunction query 2002."

[0241] Additionally, in some embodiments, there may be a microscope diagnostic prompt 2008. In various embodiments, the microscope diagnostic prompt 2008 may be one or more unstructured or plain text sentences or sentence fragments that request or command a self-diagnostic protocol or test of the charged particle microscope 302 to be run or performed (e.g., an automatic lens or aperture functionality test, an automatic electron gun functionality test). As a non-limiting example, for any suitable positive integer r, the microscope diagnostic prompt 2008 may be the following sentence: "The specimen is characterized by an image 402 or energy spectrum 404. The microscope is characterized by its current health state 406. Instruct the microscope to run the r most relevant self-diagnostic checks to probe for the malfunction specified in the natural language malfunction query 2002."

[0242] Thus, in various instances, the context component 326 can electronically execute the LLM 310 against the natural language malfunction query 2002, against the image 402, against the energy spectrum 404, against the current health state 406, against the microscope diagnostic prompt 2008, or against any suitable combination thereof. In various cases, such execution can cause the LLM 310 to generate a set of microscope self-diagnosis function calls 2006. More specifically, the context component 326 can concatenate together the natural language malfunction query 2002, the image 402, the energy spectrum 404, the current health state 406, the microscope diagnostic prompt 2008, or any combination thereof, and the context component 326 can feed the concatenation to an input layer of the encoder portion 312. In various aspects, the concatenation can complete a forward pass through one or more hidden layers of the encoder portion 312. In various instances, the output layer of the encoder portion 312 can compute or otherwise calculate one or more embeddings (not shown) based on activation or feature maps provided by one or more hidden layers of the encoder portion 312. In various cases, those one or more embeddings can be routed to the input layer of the synthesizer portion 314. In various aspects, those one or more embeddings can complete a forward pass through one or more hidden layers of the synthesizer portion 314, and the output layer of the synthesizer portion 314 can compute or otherwise calculate a set of microscope self-diagnosis function calls 2006 based on the activation or feature maps provided by the one or more hidden layers of the synthesizer portion 314.

[0243] In various instances, as described above, the microscope diagnostic prompt 2008 may request or command the charged particle microscope 302 to perform r distinct self-diagnostic protocols, tests, or checks. Accordingly, the set of microscope self-diagnostic function calls 2006 may include r calls, i.e., microscope self-diagnostic function call 2006(1) through microscope self-diagnostic function call 2006(r). In various instances, each of the set of microscope self-diagnostic function calls 2006 may be considered to be one or more respective lines of composed computer code that invokes or activates a respective self-diagnostic check or test (e.g., in any syntax understood by the charged particle microscope 302, such as JSON code or Python code) that is readable by and executable by the charged particle microscope 302. As a non-limiting example, microscope self-diagnostic function call 2006(1) may be one or more first lines of composed computer code that invokes a first self-diagnostic check (e.g., an autofocus lens self-check protocol) that may be run on charged particle microscope 302. As another non-limiting example, microscope self-diagnostic function call 2006(r) may be one or more rth lines of composed computer code that define an rth self-diagnostic check (e.g., an automatic gas injector self-check protocol) that may be run on charged particle microscope 302.

[0244] 21. In various embodiments, the context component 326 can electronically command, instruct, or otherwise cause the charged particle microscope 302 to run, perform, or execute each of the set of microscope self-diagnostic function calls 2006, thereby producing a set of self-diagnostic results 2102.

[0245] As a non-limiting example, the context component 326 may cause the charged particle microscope 302 to execute the microscope self-diagnostic function call 2006(1). That is, the context component 326 may cause the charged particle microscope 302 to automatically initiate, perform, or execute a first self-diagnostic check defined by the microscope self-diagnostic function call 2006(1). In various aspects, such a first self-diagnostic check may result in a self-diagnostic result 2102(1), which may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, or any suitable combination thereof, that the charged particle microscope 302 measures or returns during or in response to the performance of the first self-diagnostic check.

[0246] As another non-limiting example, the context component 326 may cause the charged particle microscope 302 to execute the microscope self-diagnosis function call 2006(r). That is, the context component 326 may cause the charged particle microscope 302 to automatically initiate, perform, or execute the rth self-diagnosis check defined by the microscope self-diagnosis function call 2006(r). As above, such rth self-diagnosis check may result in a self-diagnosis result 2102(r), which may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, or any suitable combination thereof, that the charged particle microscope 302 measures or returns during or in response to the performance of the rth self-diagnosis check.

[0247] In various cases, the self-diagnosis results 2102(1) through 2102(r) can be collectively considered as a set of self-diagnosis results 2102.

[0248] Now consider Figure 22. In various aspects, the model component 328 can electronically execute the LLM 310 against a concatenation of the natural language malfunction query 2002, the image 402, the energy spectrum 404, the current health state 406, the set of related documents 702, the set of inference task results 704, or the set of simulation results 2102, such execution causing the LLM 310 to generate a natural language response 1302 or synthesized code 1304.

[0249] In such an embodiment, the natural language response 1302 may include a malfunction diagnosis 2202. In various aspects, the malfunction diagnosis 2202 may be one or more unstructured or plain text declarative sentences or sentence fragments that semantically answer the natural language malfunction query 2002. That is, the malfunction diagnosis 2202 may be synthesized text that describes, states, or explains what is likely causing the charged particle microscope 302 to experience the particular microscope malfunction symptom specified in the natural language malfunction query 2002. In various instances, the sample description 2004 may be considered known information about the currently loaded sample 306. In contrast, the image 402 or energy spectrum 404 (or any of their supplemental or contextual information, such as 702, 704, 706, or 2102) may be considered measured or detected information about the currently loaded sample 306. In some cases, inconsistencies, discrepancies, or other correlations between the known information and the measured or detected information can signal or otherwise suggest what may be causing a particular problem with the charged particle microscope 302. Thus, the LLM 310 can be thought of as leveraging such inconsistencies, discrepancies, or correlations to infer or predict the cause of a particular malfunction symptom specified in the natural language malfunction query 2002. In various instances, the malfunction diagnosis 2202 can not only describe or explain the cause of any malfunction symptom specified in the natural language malfunction query 2002, but also describe or explain what sequence of steps should be taken to resolve such malfunction symptom.

[0250] As a non-limiting example, assume that the natural language malfunction query 2002 asks why the charged particle microscope 302 is producing blurry images of an organic sample. Further, assume that the organic sample must first be dehydrated before analysis by the charged particle microscope to avoid the occurrence of condensation artifacts. As discussed above, the sample description 2004 in this non-limiting example may indicate that the currently loaded sample 306 is organic. Because the LLM 310 may be trained with such information, the LLM 310 may infer or predict that the currently loaded sample 306 needs to be dehydrated before microscopic analysis. Here, in some cases, the image 402 or energy spectrum 404 may typically, frequently, or otherwise frequently contain artifacts or anomalies associated with lens condensation. Thus, the LLM 310 may determine or conclude that the currently loaded sample 306 was not subjected to the required dehydration of the sample 306 prior to analysis. Thus, the LLM 310 can accordingly synthesize a malfunction diagnosis 2202 that states or explains that failure to dehydrate the currently loaded specimen 306 is causing the particular microscopy malfunction specified in the natural language malfunction query 2002. In some cases, the malfunction diagnosis 2202 can further explain how such malfunction may be resolved (e.g., by wiping or drying the lens of the charged particle microscope 302 and by applying a dehydration protocol to the currently loaded specimen 306). As noted above, it should be noted that the set of related documents 702, the set of inference task results 704, the set of simulation results 706, or the set of self-diagnosis results 2102 can be considered additional, supplemental, contextual, or otherwise enriching information that aids or assists the LLM 310 in correctly or accurately synthesizing the fault diagnosis 2202 (e.g., correctly or accurately infer or predict what is causing a particular malfunction symptom or how to resolve it).

[0251] In various aspects, as described above, the LLM 310 can generate synthesized code 1304 in addition to (or in some cases instead of) the natural language response 1302. Also, as described above, the synthesized code 1304 can be one or more lines of computer code that can (when executed, run, or compiled) perform any suitable computerized action with respect to the charged particle microscope 302 or with respect to the natural language response 1302. As a non-limiting example, in a situation where the natural language response 1302 includes a malfunction diagnosis 2202, the LLM 310 can infer or predict that the sequential performance of one or more automatic actions of the charged particle microscope 302 (e.g., automatic lens drying or wiping, automatic electron gun reboot, automatic gas injector cycling) can or will resolve or correct the malfunction symptoms specified in the natural language malfunction query 2002. In such cases, execution of the synthesized code 1304 may cause the charged particle microscope 302 to perform such a sequence of automatic actions, thereby resolving the particular malfunction symptom. However, in other cases, the LLM 310 may instead infer or predict that manual intervention or service will be required to resolve the particular malfunction symptom. In such cases, execution of the synthesized code 1304 may cause such manual intervention or service to be scheduled or otherwise flagged (e.g., a maintenance request for the charged particle microscope 302 may be automatically inserted into an electronic service calendar or database).

[0252] As described above, the presenter component 330 can electronically perform any suitable action with respect to the natural language response 1302 or with respect to the synthesized code 1304, such as visually rendering or audibly playing the natural language response 1302. In some cases, the presenter component 330 can electronically execute, run, or compile the synthesized code 1304, thereby automatically resolving a particular malfunction symptom or automatically scheduling a service call to address a particular malfunction symptom. Thus, even if a user or technician is inexperienced or untrained in the field of charged particle microscopy, the user or technician can nonetheless competently address or handle a malfunction of the charged particle microscope 302. In this manner, the system 316 can be thought of as providing real-time, sample-tailored troubleshooting guidance to the user or technician.

[0253] 23 and 24 illustrate an exemplary, non-limiting flow diagram of a computer-implemented method according to one or more embodiments described herein.

[0254] Consider first Figure 23. In various embodiments, operation 2302 can include accessing (e.g., via 322) by a device operably coupled to the processor (e.g., 318) a natural language request (e.g., 2002) inquiring about a malfunction of the charged particle microscope (e.g., 302) and at least partially describing a currently loaded sample (e.g., 306) of the charged particle microscope.

[0255] In various aspects, as shown, the computer-implemented method 2300 may then proceed to operations 1504, 1506, 1508, and 1510 as described above. The computer-implemented method 2300 may then proceed to operation 1602 of the computer-implemented method 2400.

[0256] 24. In various cases, operation 1602 may be as described above, and computer-implemented method 2400 may proceed to operation 2402. In various cases, operation 2402 may include causing the device (e.g., via 326) to perform one or more self-diagnostic tests according to one or more function calls (e.g., 2006) generated by the large-scale language model (e.g., 310) based on a natural language request, an image or energy spectrum, or a current health state. In various cases, this may cause the charged particle microscope to produce one or more self-diagnostic results (e.g., 2102).

[0257] In various aspects, operation 2404 may include running, by the device (e.g., via 328), a large-scale language model against the natural language request (e.g., 2002), against the image (e.g., 402) or energy spectrum (e.g., 404), against the current health state (e.g., 406), against one or more documents (e.g., 702), against one or more inference task results (e.g., 704), against one or more simulation results (e.g., 706), or against one or more self-diagnosis results (e.g., 2102). In various cases, this may cause the large-scale language model to produce a natural language response (e.g., 1302, including 2202) that explains, describes, or instructs why or how the charged particle microscope is malfunctioning.

[0258] As shown, the computer-implemented method 2400 may then proceed to operation 1606 .

[0259] Now consider Figure 25. Rather than providing natural language instructions 308, a user or technician of the charged particle microscope 302 can instead provide a natural language sample query 2502.

[0260] In various aspects, the natural language sample query 2502 can be any suitable number of plain text or unstructured sentences or sentence fragments that request or command the identification of any suitable physical, chemical, or compositional attributes of the currently loaded sample 306. As some non-limiting examples, the natural language sample query 2502 can be any of the following: "What is the chemical composition of this sample?", "Tell me the crystal structure of this sample," or "What is the melting point of this sample?"

[0261] As noted above, the natural language sample query 2502 may be provided or entered by a user or technician of the charged particle microscope 302 via any suitable human computer interface device associated with the charged particle microscope 302 .

[0262] Also, as described above, the status component 324 may electronically acquire, receive, retrieve, or otherwise access an image 402, an energy spectrum 404, or a current health state 406 in response to receiving or accessing a natural language sample query 2502.

[0263] Furthermore, as described above, the context component 326 can electronically acquire, receive, retrieve, or otherwise access a set of inference task results 704 by running a set of available deep learning models 1002 on the image 402 or on the energy spectrum 404, respectively.

[0264] In various aspects, the context component 326 electronically acquires, receives, searches, or otherwise accesses the set of relevant documents 702 as described with respect to Figures 8 and 9, but the natural language instructions 308 may be replaced by a natural language sample query 2502.

[0265] Similarly, in various instances, the context component 326 electronically acquires, receives, retrieves, or otherwise accesses the set of simulation results 706, as described with respect to Figures 11 and 12, but the natural language instructions 308 may be replaced by a natural language sample query 2502. In some such cases, the digital twin prompt 1102 may be the following statement: "The sample is characterized by an image 402 or energy spectrum 404. The microscope is characterized by its current health state 406. Design q different experiments for the digital twin 602 to best determine the sample attributes identified in the natural language sample query 2502."

[0266] Thus, in various aspects, the model component 328 can electronically execute the LLM 310 against a concatenation of a natural language sample query 2502, an image 402, an energy spectrum 404, a current health state 406, a set of related documents 702, a set of inference task results 704, or a set of simulation results 706, such execution causing the LLM 310 to generate a natural language response 1302 or synthesized code 1304.

[0267] In such embodiments, the natural language response 1302 can include a sample description 2504. In various aspects, the sample description 2504 can be one or more unstructured or plain text declarative sentences or sentence fragments that semantically answer the natural language sample query 2502. That is, the sample description 2504 can be synthesized text that describes, states, or lists (e.g., as a list) any physical, chemical, or compositional attributes of the currently loaded sample 306 requested by the natural language sample query 2502. Indeed, as described above, the image 402 or energy spectrum 404 can be considered to convey at least some information regarding the chemical, physical, or compositional properties of the currently loaded sample 306. Because the LLM 310 can be tuned on the image 402 or energy spectrum 404, the LLM 310 can be considered to have access to that chemical, physical, or compositional information and can leverage that information to infer or predict an answer to the natural language sample query 2502. Thus, the LLM 310 can synthesize a sample description 2504 to state, describe, or enumerate any properties, characteristics, or attributes of the currently loaded sample 306 requested by the natural language sample query 2502. It should be noted that, as noted above, the set of related documents 702, the set of inference task results 704, or the set of simulation results 706 can be considered additional, supplemental, contextual, or otherwise enriching information that aids or assists the LLM 310 in correctly or accurately synthesizing the sample description 2504 (e.g., correctly or accurately inferring or predicting requested attributes of the currently loaded sample 306).

[0268] In various aspects, as described above, the LLM 310 can generate synthesized code 1304 in addition to (or in some cases instead of) the natural language response 1302. Also, as described above, the synthesized code 1304 can be one or more lines of computer code that can (when executed, run, or compiled) perform any suitable computerized action with respect to the charged particle microscope 302 or with respect to the natural language response 1302. As a non-limiting example, in situations where the natural language response 1302 includes a sample description 2504, execution of the synthesized code 1304 can cause the electronic display of the charged particle microscope 302 (or any suitable computerized workstation associated with the charged particle microscope 302) to display or render any suitable plot, table, or other graphic that shows any properties, characteristics, or attributes of the currently loaded sample 306 that the LLM 310 has measured or inferred.

[0269] As described above, the presenter component 330 can electronically perform any suitable action with respect to the natural language response 1302 or with respect to the synthesized code 1304, such as visually rendering or audibly playing the natural language response 1302. In some cases, the presenter component 330 can electronically execute, run, or compile the synthesized code 1304, thereby displaying various charts, tables, graphs, or plots related to the currently loaded sample 306. Thus, even if a user or technician is inexperienced or untrained in the field of charged particle microscopy, the user or technician can nonetheless utilize the charged particle microscope 302 to competently analyze the currently loaded sample 306. In this manner, the system 316 can be considered to provide a user or technician with real-time automated sample analysis.

[0270] 26 and 27 illustrate an exemplary, non-limiting flow diagram of a computer-implemented method according to one or more embodiments described herein.

[0271] Consider first Figure 26. In various embodiments, operation 2602 can include accessing (e.g., via 322) by a device operably coupled to a processor (e.g., 318) a natural language request (e.g., 2502) inquiring about one or more unknown properties of a currently loaded sample (e.g., 306) of a charged particle microscope (e.g., 302).

[0272] In various aspects, as shown, the computer-implemented method 2600 may proceed to operations 1504, 1506, 1508, and 1510 as described above. The computer-implemented method 2600 may then proceed to operation 1602 of the computer-implemented method 2700.

[0273] 27. In various cases, operation 1602 may be as described above, and the computer-implemented method 2700 may proceed to operation 2702. In various cases, operation 2702 may include executing, by the device (e.g., via 328), a large-scale language model (e.g., 310) against the natural language request (e.g., 1702), against the image (e.g., 402) or energy spectrum (e.g., 404), against the current health state (e.g., 406), against one or more documents (e.g., 702), against one or more inference task results (e.g., 704), or against one or more simulation results (e.g., 706). In various cases, this may cause the large-scale language model to produce a natural language response (e.g., 1302, including 2504) that identifies, explains, or describes one or more unknown properties of the currently loaded specimen.

[0274] As shown, the computer-implemented method 2600 may then proceed to operation 1606 .

[0275] 28. In various aspects, a user or technician of the charged particle microscope 302 may have previously provided (e.g., typed or spoken) a plurality of past natural language queries 2802 to the charged particle microscope 302. In various instances, the plurality of past natural language queries 2802 may include any suitable number of past natural language queries. In various instances, the past natural language queries may be any suitable unstructured or plain text query as described herein (e.g., may be instructions to set any of 304 to a particular value, may be a workflow query, may be a malfunction query, may be a sample query).

[0276] As described above, the status component 324 may electronically acquire, receive, retrieve, or otherwise access the image 402, energy spectrum 404, or current health state 406.

[0277] Also, as described above, the context component 326 can electronically acquire, receive, retrieve, or otherwise access a set of inference task results 704 by running a set of available deep learning models 1002 on the image 402 or on the energy spectrum 404, respectively.

[0278] Furthermore, as described above, the context component 326 can electronically acquire, receive, search, or otherwise access the set of related documents 702 by performing an appropriate embedded search in the document repository 902 (e.g., by replacing 308 with 2802 in FIG. 8).

[0279] Also, as described above, the context component 326 can use the LLM 310 to electronically obtain, receive, retrieve, or otherwise access the set of simulation results 706 by generating the appropriate function calls to the digital twin 602 (e.g., by replacing 308 with 2802 in FIG. 11 ).

[0280] Here, in various aspects, there may be a GUI prompt environment 2804. In various instances, the GUI prompt 2804 may be one or more unstructured or plain text sentences or sentence fragments that request or command that a graphical user interface (GUI) be constructed for the charged particle microscope 302 that is suitable for both the user or technician and the currently loaded sample 306. As a non-limiting example, the GUI prompt 2804 may be the following sentence: "The sample is characterized by an image 402 or energy spectrum 404. The microscope is characterized by a current health state 406. Create a microscope GUI that is relevant to the sample and suitable for the user who requested multiple previous natural language queries 2802."

[0281] Thus, in various aspects, the model component 328 can electronically execute the LLM 310 against a concatenation of multiple past natural language queries 2802, images 402, energy spectra 404, current health states 406, a set of related documents 702, a set of inference task results 704, or a set of simulation results 706, which execution can cause the LLM 310 to generate synthesized code 1304 (in some embodiments, the natural language response 1302 can be omitted).

[0282] In various aspects, the composed code 1304 may be one or more lines of computer code that can (when executed, run, or compiled) perform any suitable computerized action with respect to the charged particle microscope 302. As a non-limiting example, the composed code 1304 may define or otherwise function as source code or programming scripts for a sample-adapted GUI and a user-adapted GUI of the charged particle microscope 302.

[0283] In particular, various ones of the plurality of configurable operational settings 304 may be relevant or appropriate for some types of samples but not relevant or appropriate for other types of samples. For example, the range of possible microscopy protocols or settings that can be non-destructively or effectively applied to inorganic samples may be different or otherwise not identical to the range of possible microscopy protocols or settings that can be non-destructively or effectively applied to organic samples. As another example, the range of possible microscopy protocols or settings that can be non-destructively or effectively applied to metal samples may be different or otherwise not identical to the range of possible microscopy protocols or settings that can be non-destructively or effectively applied to plastic samples. In either case, the image 402 or energy spectrum 404 can be considered to convey at least some physical, chemical, or compositional information about the currently loaded sample 306. As the LLM 310 can be tuned on the image 402 or energy spectrum 404, the LLM 310 can use that information to infer or predict which of the plurality of configurable operational settings 304 are relevant or appropriate for the currently loaded sample 306, and which are not. Thus, the LLM 310 can create the composed code 1304 such that the composed code 1304 defines or creates a GUI that includes or displays only those of the plurality of configurable operational settings 304 that are relevant or appropriate for the currently loaded sample 306, and does not include or display any of the plurality of configurable operational settings 304 that are not relevant or appropriate for the currently loaded sample 306. In this manner, the GUI defined by the composed code 1304 can be considered adapted or customized to the currently loaded sample 306.

[0284] Additionally, various of the plurality of configurable operational settings 304 may be overly advanced or complex for an inexperienced, untrained, or unskilled user or technician. In various cases, the plurality of past natural language queries 2802 can be considered to implicitly convey or capture the user's or technician's microscopy skill level. For example, a history of asking more basic or simple queries may indicate that the user or technician has lower microscopy skills, whereas a history of asking more complex or advanced queries may indicate that the user or technician instead has higher microscopy skills. Because the LLM 310 can be tuned with the plurality of past natural language queries 2802, the LLM 310 can use that implicit information to infer or predict the user's or technician's microscopy skill or experience level. Thus, the LLM 310 can identify which of the plurality of configurable operational settings 304 are commensurate or appropriate for the inferred microscopy skill or experience level, and which are too advanced for the inferred microscopy skill or experience level. Thus, the LLM 310 can create the composed code 1304 such that the composed code 1304 defines or creates a GUI that includes or displays only those of the plurality of configurable operational settings 304 that are appropriate or less complex for the user's or technician's inferred microscopy skill or experience level, and does not include or display any of the plurality of configurable operational settings 304 that are inappropriate or too complex for the user's or technician's inferred microscopy skill or experience level. In this manner, the GUI defined by the composed code 1304 can be considered tailored or customized to the user or technician.

[0285] Furthermore, given the current state of health 406, various ones of the plurality of configurable operational settings 304 may not currently be safely invoked, activated, or altered. For example, in some cases, some components of the charged particle microscope 302 may be too worn to properly perform certain protocols or to safely adjust certain settings. As the LLM 310 may be adjusted for the current state of health 406, the LLM 310 may infer or predict which ones of the plurality of configurable operational settings 304 are not currently safely invoked, activated, or altered. Thus, the LLM 310 may create the composed code 1304 such that the composed code 1304 defines or creates a GUI that includes or displays only those of the plurality of configurable operational settings 304 that are currently safely invoked, activated, or altered, and does not include or display any of the plurality of configurable operational settings 304 that are not currently safely invoked, activated, or altered. In this manner, the GUI defined by the synthesized code 1304 can be considered adapted or customized based on the current health of the charged particle microscope 302 .

[0286] As noted above, the presenter component 330 can electronically perform any suitable action with respect to the composed code 1304. In some cases, the presenter component 330 can electronically execute, run, or compile the composed code 1304, thereby rendering or activating a GUI that is adapted or customized for the charged particle microscope 302. Thus, a wide variety of users or technicians can intuitively or comfortably interact with the charged particle microscope 302, regardless of their unique or disparate microscopy skill levels.

[0287] 29 and 30 illustrate an exemplary, non-limiting flow diagram of a computer-implemented method according to one or more embodiments described herein.

[0288] Consider first Figure 29. In various embodiments, operation 2902 can include accessing (e.g., via 322) by a device operably coupled to the processor (e.g., 318) a plurality of past natural language requests (e.g., 2802) asked by a user of the charged particle microscope (e.g., 302).

[0289] In various aspects, operation 2904 can include causing the device (e.g., via 324) to cause the charged particle microscope to capture an image (e.g., 402) or energy spectrum (e.g., 404) of the currently loaded sample (e.g., 306) using a default microscopy protocol.

[0290] In various cases, operation 2906 may include accessing, by the device (e.g., via 324), the current health state (e.g., 406) of the charged particle microscope from a digital twin (e.g., 602) that is synchronized with the charged particle microscope.

[0291] In various cases, operation 2908 can include identifying, by the device (e.g., via 326) and via embedded searching of a document database (e.g., 902), one or more documents (e.g., 702) related to multiple past natural language requests, to images or energy spectra, or to a current health condition.

[0292] In various aspects, the computer-implemented method 2900 may proceed to operation 1510, as described above, and then to operation 3002 of the computer-implemented method 3002.

[0293] Now consider Figure 30. In various embodiments, operation 3002 can include operating the digital twin by the device (e.g., via 326) according to one or more function calls (e.g., 1104) generated by a large-scale language model (e.g., 310) based on multiple past natural language requests, on images or energy spectra, or on a current health state. In various cases, this can produce one or more simulation results (e.g., 706) in the digital twin.

[0294] In various aspects, operation 3004 may include running, by the device (e.g., via 328), a large-scale language model against multiple past natural language requests, against an image or energy spectrum, against a current health state, against one or more documents, against one or more inference task results, or against one or more simulation results. In various cases, this may cause the large-scale language model to produce synthesized code (e.g., 1304) that defines a graphical user interface for the charged particle microscope adapted to the currently loaded sample and also adapted to the inferred experience level of the user.

[0295] In various cases, operation 3006 may include executing the composed code by the device (e.g., via 330). In various cases, this may activate or otherwise present a graphical user interface to the user.

[0296] According to various embodiments described herein, the LLM 310 (and the set of available deep learning models 1002) can first be trained to generate accurate or correct natural language responses 1302 or synthesized code 1304. A non-limiting example of such training is described with respect to FIG.

[0297] FIG. 31 illustrates an exemplary, non-limiting block diagram showing how the LLM 310 (or each of the set of available deep learning models 1002) may be trained, according to one or more embodiments described herein.

[0298] In various aspects, before training begins, the trainable internal parameters (e.g., convolution kernels, weight matrices, bias values) of the LLM 310 (or any of the set of deep learning models 1002) may be initialized by the system 316 in any suitable manner (e.g., via random initialization).

[0299] In various embodiments, there may be training input 3102 and ground truth annotations 3104. When it is desired to train the LLM 310, the training input 3102 may be any suitable textual, numerical, or graphical data that may be received by the LLM 310 as described herein. As merely some non-limiting examples, training input 3102 may be any suitable natural language instruction, question, command, or query (such as, for example, any of 308, 1702, 2002, 2502, or 2802), which may or may not be coupled with any suitable image (such as, for example, 402), any suitable energy spectrum (such as, for example, 404), any suitable microscope health status (such as, for example, 406), any suitable related documentation (such as, for example, 702), any suitable inference task result (such as, for example, 704), any suitable digital twin simulation result (such as, for example, 706), any suitable microscope self-diagnosis result (such as, for example, 2102), or any suitable supplemental prompt (such as, for example, 1102, 1401, 2008, or 2804). In such cases, the ground truth annotations 3104 can be any correct or accurate synthesized text content (e.g., 1302, etc.) or code (e.g., 1304, etc.) that is known or deemed to correspond to the training inputs 3102. Alternatively, when it is desired to train any of a set of available deep learning models 1002, the training inputs 3102 can be any suitable training images or training energy spectra (or a concatenation thereof), and the ground truth annotations 3104 can be any correct or accurate inference task results (e.g., correct or accurate classification labels, correct or accurate segmentation masks, correct or accurate regression outputs) that are known or deemed to correspond to the training inputs 3102.

[0300] In either case, the system 316 may cause the LLM 310 (or any of the set of available deep learning models 1002) to run on training inputs 3102, thereby causing the LLM 310 (or any of the set of available deep learning models 1002) to generate outputs 3106. More specifically, in some cases, the training inputs 3102 may be fed or routed to an input layer of the LLM 310 (or any of the set of available deep learning models 1002), the training inputs 3102 may complete a forward pass through one or more hidden layers of the LLM 310 (or any of the set of available deep learning models 1002), and the output layer of the LLM 310 (or any of the set of available deep learning models 1002) may calculate outputs 3106 based on activation maps or feature maps provided by the one or more hidden layers of the LLM 310 (or any of the set of available deep learning models 1002).

[0301] It should be noted that the format, size, or dimensionality of the output 3106 may depend on the number, arrangement, size, or other characteristics of the neurons, convolution kernels, attention blocks, or other internal parameters of the output layer (or any other layer) of the LLM 310 (or any of the set of available deep learning models 1002). Thus, the output 3106 may be forced to have any desired format, size, or dimensionality by adding, removing, or otherwise adjusting the characteristics of the output layer (or any other layer) of the LLM 310 (or any of the set of available deep learning models 1002).

[0302] In various aspects, if the output 3106 is generated by the LLM 310, the output 3106 can be considered to be predicted or inferred textual content (e.g., predicted or inferred natural language responses, predicted or inferred synthesized code, predicted or inferred function calls) that the LLM 310 synthesizes based on the training input 3102. Alternatively, if the output 3106 is generated by any of a set of available deep learning models 1002, the output 3106 can be considered to be predicted or inferred inference task results (e.g., predicted or inferred classification labels, predicted or inferred segmentation masks, predicted or inferred regression outputs) that such available deep learning models generate for the training input 3102. In either case, the ground truth annotations 3104 can be considered to be any correct or accurate results that are known or deemed to correspond to the training input 3102. Note that if the LLM 310 (or any of the set of available deep learning models 1002) has had no or little prior training, the output 3106 may be very inaccurate. In other words, the output 3106 may be very different from the ground truth annotations 3104.

[0303] In various aspects, an error 3108 (e.g., MAE, MSE, cross-entropy error) between the output 3106 and the ground truth annotations 3104 may be calculated by the system 316. In various instances, trainable internal parameters of the LLM 310 (or any of a set of available deep learning models 1002) may be incrementally updated via backpropagation (e.g., stochastic gradient descent) based on the error 3108.

[0304] In various cases, such a running and updating procedure may be repeated for any suitable number of input-annotation pairs. This may ultimately result in the iterative optimization of trainable internal parameters of the LLM 310 (or set of available deep learning models 1002) to accurately perform text or code synthesis (or any other suitable inference task, such as image or spectral classification, segmentation, or regression). In various aspects, any suitable training batch size, any suitable error / loss function, or any suitable training termination criterion may be utilized during such training.

[0305] Although the disclosure herein primarily describes the LLM 310 (or set of available deep learning models 1002) as being trained in a supervised manner, this is merely a non-limiting example for ease of explanation and illustration. In various embodiments, any other suitable training paradigm can be used to train the LLM 310 or set of available deep learning models 1002, such as unsupervised training or reinforcement learning, either of which may be associative or non-associative.

[0306] While the disclosure herein primarily describes the LLM 310 as being trained or configured to synthesize a natural language response 1302 or synthesized code 1304, these are merely non-limiting examples for ease of illustration and explanation. In various embodiments, the LLM 310 may be configured to synthesize any suitable visual graphics that can supplement the natural language response 1302. Non-limiting examples of such visual graphics may include formatted reports or presentation slides that are based on, derived from, or detailing the natural language response 1302, or plots, graphs, or charts that are based on, derived from, or detailing the natural language response 1302. Accordingly, the presenter component 330 may visually render such visual graphics in addition to or otherwise in conjunction with the natural language response 1302.

[0307] Although the disclosure herein primarily describes the presenter component 330 as visually or audibly rendering or presenting the natural language response 1302 (or any associated visual graphic) to a user or technician of the charged particle microscope 302, these are merely non-limiting examples for ease of explanation and illustration. In various embodiments, the presenter component 330 can electronically transmit the natural language response 1302 (or any associated visual graphic or composed code 1304) to any suitable computing device associated with the charged particle microscope. As a non-limiting example, the presenter component 330 can share the natural language response 1302 (or any associated visual graphic or composed code 1304) with any suitable downstream software tool or application operating for or in conjunction with the charged particle microscope 302 (e.g., some embodiments can involve sending the natural language response 1302 to such downstream software tool or application rather than presenting the natural language response 1302 to a user or technician).

[0308] Although the disclosure herein primarily describes LLM 310 as receiving natural language input (e.g., 308, 1702, 2002, 2502, 2802) provided by a user or technician of charged particle microscope 302, these are merely non-limiting examples for ease of explanation and illustration. In various embodiments, LLM 310 can receive natural language input synthesized by any suitable upstream generative model (e.g., in some cases, any of 308, 1702, 2002, 2502, or 2802 can be synthesized by an upstream generative artificial intelligence model rather than being manually provided by a user or technician).

[0309] It should be appreciated that multiple natural language inputs (e.g., multiple instances of 308, 1702, 2002, 2502, or 2802) may be provided to the system 316 (e.g., by a user or operator of the charged particle microscope 302 or by any upstream generative artificial intelligence model). In some aspects, the system 316 may utilize such multiple natural language input LLM 310 to synthesize a respective response to each of such multiple natural language inputs in the order in which such multiple natural language inputs are received. However, in other aspects, the system 316 may instead utilize the LLM 310 described herein to synthesize a respective response to each of such multiple natural language inputs in any other suitable order (e.g., in an order different from the order in which the multiple natural language inputs are received). In some cases, such multiple natural language inputs may be provided to the system 316 simultaneously or otherwise substantially simultaneously with one another such that they may be considered to collectively form a single overarching prompt. It will be appreciated that in such situations, the multiple natural language inputs may be individually identified through the application of any suitable prompt subdivision or prompt segmentation techniques to such a single overarching prompt.

[0310] While the disclosure herein primarily describes the system 316 as utilizing the LLM 310 to respond to natural language input (e.g., 308, 1702, 2002, 2502, 2802) associated with the charged particle microscope 302, these are merely non-limiting examples for ease of explanation and illustration. In various embodiments, the system 316 can utilize the LLM 310 to periodically or periodically monitor the status or context of the charged particle microscope 302 even in the absence of any natural language input (e.g., 308, 1702, 2002, 2502, and 2802). As a non-limiting example, in some embodiments, the status component 324 can electronically acquire an image 402, an energy spectrum 404, or a current health state 406 (e.g., via a weight sensor or a pressure sensor) in response to determining that an operational stage of the charged particle microscope 302 is not free. In such a case, the context component 326 may electronically acquire, receive, search, or otherwise access a set of inference task results 704 by executing a set of available deep learning models 1002 on the image 402 or on the energy spectrum 404, respectively. Also, in such a case, the context component 326 may electronically acquire, receive, search, or otherwise access a set of related documents 702 as described with respect to Figures 8 and 9, but the natural language instructions 308 may be omitted. Furthermore, in such a case, the context component 326 may electronically acquire, receive, search, or otherwise access a set of simulation results 706 as described with respect to Figures 11 and 12, but the natural language instructions 308 may be omitted.Thus, the model component 328 can electronically execute the LLM 310 on a concatenation of an image 402, an energy spectrum 404, a current health state 406, a set of related documents 702, a set of inference task results 704, or a set of simulation results 706, and such execution can cause the LLM 310 to generate a natural language response 1302 or synthesized code 1304.

[0311] In such embodiments, the natural language response 1302 may include one or more unstructured or plain text declarative sentences or sentence fragments that are semantically related in any suitable manner to the image 402, the energy spectrum 404, the current health state 406, the set of related documents 702, the set of inference task results 704, or the set of simulation results 706. In some aspects, the natural language response 1302 in such cases may be synthesized text that describes or explains any suitable step, action, or task (as inferred by the LLM 310) that is recommended to be performed (e.g., by a user or technician, or by the charged particle microscope 302 itself) given the state (e.g., 402, 404, or 406) or context (e.g., 702, 704, or 706) of the charged particle microscope 302.

[0312] As a non-limiting example, the LLM 310 may infer that a particular microscopy workflow or protocol should be performed on the currently loaded sample 306 (e.g., to avoid sample degradation or imaging artifacts). As such, the natural language response 1302 may explain or describe how to perform such a particular microscopy workflow or protocol without any query or command by a user or technician of the charged particle microscope 302. In some instances, the synthesized code 1304 may be configured to automatically perform all or part of that particular microscopy workflow or protocol.

[0313] As another non-limiting example, the LLM 310 may infer that a particular microscopy workflow or protocol should be performed on the currently loaded sample 306 (e.g., to avoid sample degradation or imaging artifacts), but the LLM 310 may also infer that a critical component (e.g., an x-ray tube) of the charged particle microscope 302 has too much wear to properly or safely perform such a particular microscopy workflow or protocol. As such, the natural language response 1302 may explain such inference and recommend that the critical component be serviced despite the absence of any query or command by a user or technician of the charged particle microscope 302. In some instances, the synthesized code 1304 may be configured to automatically schedule such servicing.

[0314] Various embodiments described herein may be considered to utilize large-scale language models to improve the accessibility or ease of operation of charged particle microscopes. In particular, various embodiments may involve adjusting an LLM with an image or energy spectrum captured by the charged particle microscope. Such adjustment may enable the LLM to synthesize text content or code for the charged particle microscope more informed or accurately than would otherwise be the case. In other words, the text or code synthesized by the LLM would be less accurate or less satisfying to a user or technician of the charged particle microscope if the LLM were to receive as input a text prompt that was not accompanied by an image or energy spectrum captured by the charged particle microscope. Thus, image or spectrum adjustment of an LLM as described herein may be considered a concrete, tangible improvement in the field of charged particle microscope operation.

[0315] The scientific instrument systems, methods, or techniques disclosed herein may involve interactions with a human user (e.g., via a user local computing device 3420 discussed herein with reference to FIG. 34). These interactions may include providing the user with information (e.g., information about the operation of the scientific instrument, such as the scientific instrument 3410 of FIG. 34, information about a sample being analyzed or other tests or measurements performed by the scientific instrument, information retrieved from a local or remote database, or other information), or providing options for the user to enter commands (e.g., to control the operation of the scientific instrument, such as the scientific instrument 3410 of FIG. 34, or to control the analysis of data generated by the scientific instrument), queries (e.g., to a local or remote database), or other information. In some embodiments, these interactions may be performed via a graphical user interface (GUI) that includes a visual display on a display device (e.g., display device 3310 discussed herein with reference to FIG. 33) that provides output to a user and / or prompts the user to provide input (e.g., via one or more input devices such as a keyboard, mouse, trackpad, or touchscreen included in other I / O devices 3312 discussed herein with reference to FIG. 33). The scientific instrument systems, methods, or techniques disclosed herein may include any suitable GUI for interaction with a user.

[0316] 32 depicts an exemplary graphical user interface 3200 (hereinafter, "GUI 3200") that may be used in implementing some or all of the support methods or techniques disclosed herein, according to various embodiments. In various aspects, the GUI 3200 may be provided on any suitable electronic display (e.g., display device 3310 as discussed herein with reference to FIG. 33) of a computing device (e.g., computing device 3300 as discussed herein with reference to FIG. 33) of a scientific instrument support system (e.g., scientific instrument support system 3400 as discussed herein with reference to FIG. 34), and a user or technician may interact with the GUI 3200 using any suitable input device (e.g., any of the other I / O devices 3312 as discussed herein with reference to FIG. 33) and input technique (e.g., cursor movement, motion capture, facial recognition, gesture detection, voice recognition, button actuation).

[0317] GUI 3200 can include a data display area 3202, a data analysis area 3204, a scientific instrument control area 3206, and a settings area 3208. The particular number and arrangement of areas depicted in Figure 32 is merely illustrative, and other embodiments of GUI 3200 can include any number and arrangement of areas, including any desired features.

[0318] The data display area 3202 can display data generated by a scientific instrument (eg, the scientific instrument 3410 discussed herein with reference to FIG. 34).

[0319] Data analysis area 3204 can display any suitable data analysis results (e.g., the results of analyzing the data illustrated in data display area 3202 or other data). In some embodiments, data display area 3202 and data analysis area 3204 can be combined within GUI 3200 (e.g., to include both data output from a scientific instrument and some analysis of the data in a common graph or area).

[0320] The scientific instrument control area 3206 may include options that allow a user or technician to control scientific instruments (e.g., scientific instruments 3410 discussed herein with reference to FIG. 34). For example, the scientific instrument control area 3206 may include configurable parameters that govern the operation of such scientific instruments (e.g., configurable parameters that govern the voltage or current of the scientific instrument, manage the internal temperature of the scientific instrument, or govern the fluid flow rate of the scientific instrument).

[0321] Settings area 3208 may include options that enable a user or technician to control any feature or function of GUI 3200 (or of other GUIs) or perform common computing operations with respect to data display area 3202 and data analysis area 3204 (e.g., saving data to a storage device such as storage device 3304 discussed herein with reference to FIG. 33 , sending data to another user, labeling data).

[0322] As described above, the scientific instrument module 102 may be implemented by one or more computing devices. Figure 33 is a block diagram of a computing device 3300 capable of implementing some or all of the scientific instrument methods or techniques disclosed herein, according to various embodiments. In some embodiments, the scientific instrument module 102 may be implemented by a single instance of the computing device 3300 or by multiple instances of the computing device 3300. Furthermore, as discussed below, the computing device 3300 (or multiple instances thereof) implementing the scientific instrument module 102 may be part of one or more of the scientific instrument 3410, user local computing device 3420, service local computing device 3430, or remote computing device 3440 of Figure 34.

[0323] While computing device 3300 is illustrated as having several components, any one or more of these components may be omitted or duplicated as suitable for the application and configuration. In some embodiments, some or all of the components included in computing device 3300 may be mounted on one or more motherboards and enclosed in a housing (e.g., comprising plastic, metal, or other material). In some embodiments, several of these components may be fabricated on a single system-on-a-chip (SoC) (e.g., an SoC may include one or more instances of processing device 3302 and one or more instances of storage device 3304). 33, but may include interface circuitry (not shown) for coupling to the one or more omitted components using any suitable interface (e.g., a Universal Serial Bus (USB) interface, a High-Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface). For example, the computing device 3300 may omit the display device 3310, but may include display device interface circuitry (e.g., connectors and driver circuits) to which the display device 3310 can be coupled.

[0324] The computing device 3300 may include a processing device 3302 (e.g., one or more processing devices). As used herein, the term "processing device" may refer to any device or portion of a device that processes electronic data from registers or memory and converts the electronic data into other electronic data that may be stored in registers or memory. The processing device 3302 may include one or more digital signal processors (DSPs), application specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GSMUs), or other processors. The processing device may include a graphics processing unit (GPU), a cryptographic processor (a dedicated processor that runs cryptographic algorithms in hardware), a server processor, or any other suitable processing device.

[0325] The computing device 3300 may include a storage device 3304 (e.g., one or more storage devices). The storage device 3304 may include random access memory (RAM) (e.g., static RAM). The storage device 3304 may include one or more memory devices, such as static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive-bridging RAM (CBRAM) devices, hard-drive-based memory devices, solid-state memory devices, network drives, cloud drives, or any combination of memory devices. In some embodiments, the storage device 3304 may include memory sharing a die with the processing device 3302. In such embodiments, the memory may be used as cache memory and may include, for example, embedded dynamic random access memory (eDRAM) or spin transfer torque magnetic random access memory (STT-MRAM). In some embodiments, the storage device 3304 may include a non-transitory computer-readable medium having instructions, which when executed by one or more processing devices (e.g., processing device 3302), cause the computing device 3300 to perform any appropriate one or portions thereof of the methods disclosed herein.

[0326] The computing device 3300 may include an interface device 3306 (e.g., one or more instances of an interface device 3306). The interface device 3306 may include one or more communication chips, connectors, or other hardware and software for managing communications between the computing device 3300 and other computing devices. For example, the interface device 3306 may include circuitry for managing wireless communications for transferring data to and from the computing device 3300. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, or communications channels that can communicate data using modulated electromagnetic radiation over a non-solid medium. This term does not imply that the associated devices do not include any wiring, although in some embodiments they may not. The circuitry included in interface device 3306 for managing wireless communications may implement any of several wireless standards or protocols, including, but not limited to, Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.11 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), the Long-Term Evolution (LTE) project (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also referred to as "3GPP®2")) with any amendments, updates, and / or revisions.In some embodiments, the circuitry included in the interface device 3306 for managing wireless communications is compatible with Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA, and the like. In some embodiments, the circuitry included within the interface device 3306 for managing wireless communications may operate in accordance with a GSM HSPA, E-HSPA, or LTE network. In some embodiments, the circuitry included within the interface device 3306 for managing wireless communications may operate in accordance with an Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, the circuitry included within the interface device 3306 for managing wireless communications may operate in accordance with a Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), or LTE network. Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications Enhanced Cordless Telecommunication (DECT), Evolution-Data Optimized (EV-DO), and their derivatives, as well as any other wireless protocols designated as 3G, 4G, 5G, and beyond. In some embodiments, the interfacing device 3306 may include one or more antennas (e.g., one or more antenna arrays) for receiving and / or transmitting wireless communications.

[0327] In some embodiments, the interface device 3306 may include circuitry for managing wired communications, such as electrical, optical, or any other suitable communications protocol. For example, the interface device 3306 may include circuitry supporting communications according to Ethernet technology. In some embodiments, the interface device 3306 may support both wireless and wired communications, or may support multiple wired or multiple wireless communications protocols. For example, a first set of circuits in the interface device 3306 may be dedicated to short-range wireless communications, such as Wi-Fi or Bluetooth, and a second set of circuits in the interface device 3306 may be dedicated to long-range wireless communications, such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, or others. In some embodiments, the first set of circuits in the interface device 3306 may be dedicated to wireless communications, and the second set of circuits in the interface device 3306 may be dedicated to wired communications.

[0328] The computing device 3300 may include a battery / power circuit 3308. The battery / power circuit 3308 may include one or more energy storage devices (e.g., batteries or capacitors) or circuitry for coupling components of the computing device 3300 to an energy source remote from the computing device 3300 (e.g., an AC line power source).

[0329] The computing device 3300 may include a display device 3310 (e.g., multiple display devices). The display device 3310 may include any visual indicator such as a heads-up display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0330] The computing device 3300 may include other input / output (I / O) devices 3312. The other I / O devices 3312 may include, for example, one or more audio output devices (e.g., speakers, headsets, earphones, alarms), one or more audio input devices (e.g., microphones or microphone arrays), a location determination device (e.g., a GPS device that communicates with a satellite-based system to receive the location of the computing device 3300), an audio codec, a video codec, a printer, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, accelerometers, gyroscopes), an image capture device such as a camera, a cursor control device such as a keyboard, mouse, stylus, trackball, or touchpad, a barcode reader, a Quick Response (QR) code reader, or a radio frequency identification (RFID) device. The device may include a radio frequency identification (RFID) reader.

[0331] The computing device 3300 can have any suitable form factor for its use and configuration, such as a handheld or mobile computing device (e.g., a mobile phone, a smartphone, a mobile internet device, a tablet computer, a laptop computer, a netbook computer, an ultrabook computer, a personal digital assistant (PDA), an ultra-mobile personal computer), a desktop computing device, or a server computing device or other networked computing component.

[0332] One or more computing devices implementing any of the scientific instrument modules, methods, or techniques disclosed herein may be part of a scientific instrument support system. Figure 34 is a block diagram of an exemplary scientific instrument support system 3400 in which some or all of the scientific instrument support methods disclosed herein may be implemented, according to various embodiments. Any of the scientific instrument modules, methods, or techniques disclosed herein (e.g., scientific instrument module 102, computer-implemented method 200, system 316, computer-implemented methods 1500-1600, 1800-1900, 2300-2400, 2600-2700, or 2900-3000) may be implemented by one or more of a scientific instrument 3410, a user local computing device 3420, a service local computing device 3430, or a remote computing device 3440 of the scientific instrument support system 3400.

[0333] Any of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440 may include any of the embodiments of computing device 3300, and any of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440 may take the form of any suitable of the embodiments of computing device 3300.

[0334] The scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440 may each include a processing device 3402, a storage device 3404, and an interface device 3406. The processing device 3402 may take any suitable form, including any form of processing device 3302, and the processing devices 3402 included in different ones of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440 may take the same form or different forms. The storage device 3404 may take any suitable form, including any form of storage device 3304, and the storage devices 3404 included in different ones of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440 may take the same form or different forms. The interface device 3406 may take any suitable form, including any form of interface device 3306, and the interface devices 3406 included in different ones of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440 may take the same form or different forms.

[0335] The scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, and the remote computing device 3440 can communicate with other elements of the scientific instrument support system 3400 via communication paths 3408. The communication paths 3408 may communicatively couple the interface devices 3406 of different ones of the elements of the scientific instrument support system 3400, as shown, and may be wired or wireless communication paths (e.g., by any of the communication techniques discussed herein with reference to the interface device 3306). While the particular scientific instrument support system 3400 depicted in FIG. 34 includes communication paths between each pair of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, and the remote computing device 3440, this “fully connected” implementation is merely illustrative, and in various embodiments, various ones of the communication paths 3408 may not be present. For example, in some embodiments, the service local computing device 3430 may lack a direct communication path 3408 between the interface device 3406 of the service local computing device 3430 and the interface device 3406 of the scientific instrument 3410, but instead may communicate with the scientific instrument 3410 via a communication path 3408 between the service local computing device 3430 and the user local computing device 3420, and a communication path 3408 between the user local computing device 3420 and the scientific instrument 3410.

[0336] Scientific instrument 3410 may include any suitable scientific instrument, such as charged particle microscope 302 .

[0337] The user local computing device 3420 may be a computing device that is local to the user of the scientific instrument 3410 (e.g., according to any of the embodiments of the computing device 3300). In some embodiments, the user local computing device 3420 may also be local to the scientific instrument 3410, but need not be; for example, a user local computing device 3420 in a user's home or office may be remote from but in communication with the scientific instrument 3410 such that the user may use the user local computing device 3420 to control or access data from the scientific instrument 3410. In some embodiments, the user local computing device 3420 may be a laptop, smartphone, or tablet device. In some embodiments, the user local computing device 3420 may be a portable computing device.

[0338] The service local computing device 3430 may be a computing device (e.g., according to any of the embodiments of computing device 3300) that is local to an entity that provides services to the scientific instrument 3410. For example, the service local computing device 3430 may be local to the manufacturer of the scientific instrument 3410 or a third-party service company. In some embodiments, the service local computing device 3430 may communicate with the scientific instrument 3410, the user local computing device 3420, or the remote computing device 3440 (e.g., via a direct communication path 3408 or via multiple "indirect" communication paths 3408, as discussed above) to receive data regarding the operation of the scientific instrument 3410, the user local computing device 3420, or the remote computing device 3440 (e.g., results of self-tests of the scientific instrument 3410, calibration coefficients used by the scientific instrument 3410, measurements of sensors associated with the scientific instrument 3410). In some embodiments, the service local computing device 3430 may communicate with the scientific instrument 3410, the user local computing device 3420, or the remote computing device 3440 (e.g., via a direct communication path 3408 or via multiple "indirect" communication paths 3408, as discussed above) to transmit data to the scientific instrument 3410, the user local computing device 3420, or the remote computing device 3440 (e.g., to update programmed instructions such as firmware in the scientific instrument 3410, to initiate the performance of a test or calibration sequence in the scientific instrument 3410, to update programmed instructions such as software in the user local computing device 3420 or the remote computing device 3440).A user of the scientific instrument 3410 can utilize the scientific instrument 3410 or user local computing device 3420 to communicate with the service local computing device 3430 to report problems with the scientific instrument 3410 or user local computing device 3420, request a technician visit to improve the operation of the scientific instrument 3410, or order consumables or replacement parts related to the scientific instrument 3410 or for other purposes.

[0339] The remote computing device 3440 may be a computing device that is remote from the scientific instrument 3410 or the user local computing device 3420 (e.g., according to any of the embodiments of the computing device 3300 discussed herein). In some embodiments, the remote computing device 3440 may be included in a data center or other large-scale server environment. In some embodiments, the remote computing device 3440 may include network-attached storage (e.g., as part of the storage device 3404). The remote computing device 3440 may store data generated by the scientific instrument 3410, perform analysis of the data generated by the scientific instrument 3410 (e.g., according to programmed instructions), facilitate communications between the user local computing device 3420 and the scientific instrument 3410, or facilitate communications between the service local computing device 3430 and the scientific instrument 3410.

[0340] In some embodiments, one or more of the elements of the scientific instrument support system 3400 illustrated in Figure 34 may be omitted. Furthermore, in some embodiments, more than one of various of the elements of the scientific instrument support system 3400 of Figure 34 may be present. For example, the scientific instrument support system 3400 may include multiple user local computing devices 3420 (e.g., different user local computing devices 3420 associated with different users or at different locations). In another example, the scientific instrument support system 3400 may include multiple scientific instruments 3410 that all communicate with a service local computing device 3430 and / or a remote computing device 3440; in such an embodiment, the service local computing device 3430 may monitor these multiple scientific instruments 3410, or the service local computing device 3430 may cause updates or other information to be "broadcast" to the multiple scientific instruments 3410 simultaneously. Different scientific instruments 3410 in the scientific instrument support system 3400 can be located near each other (e.g., in the same room) or far from each other (e.g., on different floors of a building, in d...

Claims

1. 1. A system comprising: a processor executing computer-executable components stored in non-transitory computer-readable memory, the computer-executable components comprising: an access component that accesses natural language instructions associated with a charged particle microscope, the natural language instructions requesting or instructing the charged particle microscope to undergo a configurable setting adjustment or to perform an automated task; a state component that, in response to receiving the natural language command, causes the charged particle microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol; a model component that runs a large-scale language model on both the natural language instruction and the image or energy spectrum of the sample, thereby generating a natural language response that indicates how implementing the natural language instruction will affect the sample.

2. The computer-executable components include: A presenter component, visually rendering the natural language response or a visual graphic associated with the natural language response on an electronic display associated with the charged particle microscope; audibly playing the natural language response over an electronic speaker associated with the charged particle microscope; or The system of claim 1 , further comprising a presenter component that transmits the natural language response to a computing device associated with the charged particle microscope.

3. 10. The system of claim 1, wherein the natural language instructions are plain text typed into a graphical user interface text field associated with the charged particle microscope or plain text transcribed from an audio recording captured by a microphone associated with the charged particle microscope.

4. 2. The system of claim 1, wherein the natural language response indicates that performing the natural language command will harm or charge the specimen, and the natural language response further indicates that the charged particle microscope should undergo an alternative configurable setting adjustment or that the charged particle microscope should perform an alternative automated task.

5. The computer-executable components include: The system of claim 4 , further comprising a presenter component that causes the charged particle microscope to undergo the alternative configurable setting adjustment or perform the alternative automated task.

6. The computer-executable components include:

10. The system of claim 1, further comprising a context component that identifies one or more documents related to the natural language instruction and to the image or energy spectrum of the sample via an embedded search of a document repository, wherein the large-scale language model receives as input the natural language instruction, the image or energy spectrum of the sample, and the one or more documents.

7. The computer-executable components include:

10. The system of claim 1, further comprising a context component that runs one or more available deep learning models on the image or energy spectrum of the sample, thereby generating one or more inference task results, wherein the large-scale language model receives as input the natural language instruction, the image or energy spectrum of the sample, and the one or more inference task results.

8. the charged particle microscope is synchronized with the digital twin; the large-scale language model receives as input the natural language instruction, the image or energy spectrum of the sample, and the current health state of the charged particle microscope as represented by the digital twin; or 2. The system of claim 1, wherein the large-scale language model receives as input the natural language instruction, the image or energy spectrum of the sample, and one or more simulation results for the charged particle microscope generated by the digital twin.

9. 1. A computer-implemented method comprising: accessing, by a device operatively coupled to a processor, natural language instructions associated with a charged particle microscope, the natural language instructions requesting or directing the charged particle microscope to undergo a configurable setting adjustment or to perform an automated task; and in response to receiving the natural language command, by the device, causing the charged particle microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol. and executing, by the device, a large-scale language model on both the natural language instruction and the image or energy spectrum of the sample, thereby generating a natural language response indicating how executing the natural language instruction will affect the sample.

10. visually rendering, by the device, the natural language response or a visual graphic associated with the natural language response on an electronic display associated with the charged particle microscope; audibly playing, by the device, the natural language response on an electronic speaker associated with the charged particle microscope; or transmitting, by the device, the natural language response to a computing device associated with the charged particle microscope; The computer-implemented method of claim 9 further comprising:

11. 10. The computer-implemented method of claim 9, wherein the natural language instructions are plain text typed into a graphical user interface text field associated with the charged particle microscope or plain text transcribed from an audio recording captured by a microphone associated with the charged particle microscope.

12. 10. The computer-implemented method of claim 9, wherein the natural language response indicates that performing the natural language command will harm or charge the specimen, and the natural language response further indicates that the charged particle microscope should undergo an alternative configurable setting adjustment or that the charged particle microscope should perform an alternative automated task.

13. causing the device to cause the charged particle microscope to undergo the alternative configurable setting adjustment or perform the alternative automated task; The computer-implemented method of claim 12 further comprising:

14. and further comprising identifying, by the device, one or more documents related to the natural language instruction and to the image or energy spectrum of the sample via an embedded search of a document repository, wherein the large-scale language model receives as input the natural language instruction, the image or energy spectrum of the sample, and the one or more documents.

10. The computer-implemented method of claim 9.

15. and further comprising: executing, by the device, one or more available deep learning models on the image or energy spectrum of the sample, thereby generating one or more inference task results, wherein the large-scale language model receives as input the natural language instruction, the image or energy spectrum of the sample, and the one or more inference task results.

10. The computer-implemented method of claim 9.

16. the charged particle microscope is synchronized with the digital twin; the large-scale language model receives as input the natural language instruction, the image or energy spectrum of the sample, and the current health state of the charged particle microscope as represented by the digital twin; or 10. The computer-implemented method of claim 9, wherein the large-scale language model receives as input the natural language instruction, the image or energy spectrum of the sample, and one or more simulation results for the charged particle microscope generated by the digital twin.

17. 1. A computer program for facilitating large scale language model support for charged particle microscope operations, the computer program comprising: a non-transitory computer readable memory, the non-transitory computer readable memory having program instructions embodied in the non-transitory computer readable memory, the program instructions causing a processor to: accessing plain text commands provided by a user of the scanning electron microscope, the plain text commands requesting the scanning electron microscope to perform specified microscope operations; in response to receiving the plain text command, causing the scanning electron microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the scanning electron microscope via a default microscopy protocol; executing a large scale language model on both the plain text command and the image or energy spectrum of the sample, the large scale language model generating as output a plain text response indicating whether the specified microscopy operation will damage the sample; A computer program executable by the processor to cause the plain text response to be visually or audibly rendered on an electronic display or electronic speaker associated with the scanning electron microscope.

18. 20. The computer program of claim 17, wherein the plain text response indicates that the specified microscopy operation will damage the specimen, and the plain text response further indicates that such damage can be avoided by an alternative microscopy operation.

19. The program instructions may cause the processor to:

20. The computer program of claim 18, further executable to instruct the scanning electron microscope to perform the alternative microscopy operation.

20. 18. The computer program of claim 17, wherein the large-scale language model receives as input the plain text command, the image or energy spectrum of the sample, and one or more simulation results produced by a digital twin synchronized with the scanning electron microscope.

Images