Large language model assistance for charged particle microscope operation

Abstract

The invention relates to large language model assistance for charged particle microscope operation. Systems / techniques are provided for facilitating large language model assistance for charged particle microscope operation. In various embodiments, a system may access natural language instructions associated with a charged particle microscope, where the natural language instructions may request the charged particle microscope to undergo configurable setting adjustments or perform automation tasks. In various aspects, the system may cause the charged particle microscope to capture an image or energy spectrum of a sample currently loaded on a stage of the charged particle microscope according to a default microscopy protocol in response to the natural language instruction. In various instances, the system may execute a large language model on both the natural language instruction and the image or energy spectrum of the sample to produce a natural language response indicating how implementation of the natural language instruction will affect the sample.

Description

Background Technology

[0001] The field of charged particle microscopy has historically been limited by its operational complexity. This complexity can prevent users from interacting with charged particle microscopes effectively or intuitively. Summary of the Invention

[0002] The following summary is presented to provide a basic understanding of one or more embodiments. This summary is not intended to identify key or essential elements, or to depict any scope of a particular embodiment or any scope of the claims. Its sole purpose is to present the concepts in a simplified form as a prelude to the more detailed description that follows. In one or more embodiments described herein, devices, systems, computer-implemented methods, apparatuses, or computer program products that facilitate large language model assistance for operation of charged particle microscopy are described.

[0003] According to one or more embodiments, a system is provided. The system may include a non-transitory computer-readable storage memory for storing computer-executable components. The system may also include a processor operatively coupled to the non-transitory computer-readable storage memory and executable the computer-executable components stored therein. In various embodiments, these computer-executable components may include an access component that accesses natural language instructions provided by a user of a charged particle microscope, wherein the natural language instructions request or command the charged particle microscope to undergo configurable setting adjustments or perform automated tasks. In various aspects, these computer-executable components may include a state component that, in response to the receipt of a natural language instruction, causes the charged particle microscope to capture an image or energy spectrum of a sample currently mounted on the stage of the charged particle microscope according to a default microscopy protocol. In various cases, these computer-executable components may include a model component that performs a large language model on both the natural language instruction and the image or energy spectrum of the sample, thereby producing a natural language response indicating how implementing the natural language instruction will affect the sample.

[0004] According to one or more embodiments, a computer-implemented method is provided. In various embodiments, the computer-implemented method may include access, by a device operatively coupled to a processor, to natural language instructions provided by a user of a charged particle microscope, wherein the natural language instructions may request or command the charged particle microscope to undergo configurable setting adjustments or perform automated tasks. In various aspects, the computer-implemented method may include, by the device and in response to the receipt of the natural language instructions, causing the charged particle microscope to capture an image or energy spectrum of a sample currently mounted on the stage of the charged particle microscope according to a default microscopy protocol. In various cases, the computer-implemented method may include by the device performing a large language model on both the natural language instructions and the image or energy spectrum of the sample, thereby generating a natural language response indicating how implementing the natural language instructions will affect the sample.

[0005] According to one or more embodiments, a computer program product is provided for facilitating large language model assistance for operation of charged particle microscopy. In various embodiments, the computer program product may include a non-transitory computer-readable storage memory containing program instructions. In various aspects, these program instructions may be executable by a processor to enable the processor to access a plain text command provided by a user of a scanning electron microscope, wherein the plain text command requests the scanning electron microscope to perform a specified microscopy action. In various cases, these program instructions may be executable by a processor to cause the processor, in response to the receipt of the plain text command, cause the scanning electron microscope to capture an image or energy spectrum of a sample currently mounted on the stage of the scanning electron microscope via a default microscopy protocol. In various cases, these program instructions may be executable by a processor to cause the processor to perform a large language model on both the plain text command and the image or energy spectrum of the sample, wherein the large language model can produce a plain text response as output indicating whether the specified microscopy action will damage the sample. In various aspects, these program instructions may be executable by a processor to cause the processor to visually or audibly present the plain text response on an electronic display or electronic speaker associated with the scanning electron microscope. Attached Figure Description

[0006] Various embodiments will be readily understood through the following detailed description taken in conjunction with the accompanying drawings. For ease of description, the same reference numerals indicate the same structural elements. The embodiments are illustrated in the figures by way of example rather than limitation. These figures are not necessarily drawn to scale.

[0007] Figure 1 Example non-limiting block diagrams of scientific instrument modules according to various embodiments described herein are shown.

[0008] Figure 2 Example non-limiting flowcharts illustrating computer-implemented methods according to various embodiments described herein are provided.

[0009] Figure 3 A block diagram illustrating an example non-limiting system for facilitating large language model assistance for charged particle microscopy operation according to one or more embodiments described herein is shown.

[0010] Figure 4 A block diagram of an example non-limiting system for facilitating operation of charged particle microscopy with a large language model aid according to one or more embodiments described herein is illustrated, the system including a sample image, a sample energy spectrum, and the current microscope health status.

[0011] Figure 5 Example non-limiting block diagrams illustrating how sample images or energy spectra can be obtained according to one or more embodiments described herein.

[0012] Figure 6 Example non-limiting block diagrams illustrating how the current health status of a microscope can be obtained according to one or more embodiments described herein are shown.

[0013] Figure 7 A block diagram of an example non-limiting system for facilitating large language model-assisted operation of charged particle microscopy according to one or more embodiments described herein is illustrated, the system comprising a set of relevant documents, a set of inference task results, and a set of simulation results.

[0014] Figures 8 to 12 Example non-limiting block diagrams illustrating how a set of relevant documents, a set of inference task results, and a set of simulation results can be obtained according to one or more embodiments described herein.

[0015] Figure 13 A block diagram of an example non-limiting system for facilitating operation of charged particle microscopy with a large language model aid according to one or more embodiments described herein is illustrated, the system including natural language responses and synthesized code.

[0016] Figures 14 to 30 Example non-limiting block diagrams or flowcharts illustrating how natural language responses and synthesized code can be obtained according to one or more embodiments described herein.

[0017] Figure 31 Example non-limiting block diagrams illustrating how various artificial intelligence models can be trained according to one or more embodiments described herein are presented.

[0018] Figure 32Example non-limiting block diagrams illustrating graphical user interfaces that can be used to perform some or all of the methods or techniques disclosed herein, according to various embodiments described herein.

[0019] Figure 33 Example non-limiting block diagrams illustrate computing devices that can perform some or all of the methods or techniques disclosed herein, according to various embodiments described herein.

[0020] Figure 34 Examples of non-limiting block diagrams illustrating scientific instrument support systems in which some or all of the methods or techniques disclosed herein can be performed, according to various embodiments described herein.

[0021] Figure 35 A block diagram illustrating an example non-limiting operating environment in which one or more embodiments described herein may be facilitated.

[0022] Figure 36 Example networking environments are illustrated that are operable to perform the various specific implementations described herein.

[0023] Figure 37 Example dual-beam microscopes that can be implemented according to the various implementation schemes described herein are illustrated. Detailed Implementation

[0024] The following detailed description is merely illustrative and is not intended to limit the implementation and / or application or use of the embodiments. Furthermore, there is no intention to be bound by any express or implied information presented in the preceding background or invention summary or detailed description sections.

[0025] One or more embodiments will now be described with reference to the accompanying drawings, wherein the same reference numerals are used throughout to refer to the same elements. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a more thorough understanding of one or more embodiments. However, it will be apparent that in various cases, one or more embodiments may be practiced without these specific details.

[0026] The various operations can be described sequentially as multiple discrete actions or operations in a manner most conducive to understanding the subject matter disclosed herein. However, the described order should not be construed as implying that these operations must depend on the order. Specifically, these operations may be performed in an order different from the order presented. The described operations may be performed in an order different from the described embodiments. Various additional operations may be performed, or the described operations may be omitted in additional embodiments.

[0027] While some elements may be represented in the singular (e.g., "processing device"), any suitable element may be represented by multiple instances of that element, and vice versa. For example, a set of operations described as being performed by a processing device may be implemented as different operations of those operations performed by different processing devices. As used herein, the phrase "based on" should be understood to mean "at least partially based on," unless otherwise specified.

[0028] Charged particle microscopes (e.g., scanning electron microscopes (SEM), transmission electron microscopes (TEM), electron energy loss microscopes (EELM)) can be any suitable computerized device capable of capturing or generating microscopic or nanoscale images or energy spectra in scientific, laboratory, research, or clinical operating environments. To facilitate the capture or generation of such images or energy spectra, charged particle microscopes may utilize complex arrangements of actuated components (e.g., ion sources, electron sources, optical lenses or apertures, optical plates or deflectors, columns, coils, heaters, coolers, fluid valves, fluid pumps, circuit switches, sample stages), sensors (e.g., ion detectors, electron detectors, voltmeters, thermistors, potentiometers, pressure gauges), or consumables (e.g., carrier liquids, calibrators, filters, reactive gases).

[0029] The field of charged particle microscopy has historically been limited by its operational complexity. In other words, because charged particle microscopes can have such complex structures, their operation or use can be quite complex. In fact, for a user to be competent or confident in using a charged particle microscope to analyze clinical or laboratory samples, they typically require extensive specialized training, education, or certification in the field. For example, a user might learn how to properly operate the graphical user interface (GUI) or physical controls of a charged particle microscope by attending a microscope course lasting several weeks or months (after all, charged particle microscopes can appear at first glance to have a dazzling or overwhelming number of configurable software or hardware settings, buttons, knobs, sliders, or options). Users who haven't undergone such extensive training may not be able to use a charged particle microscope effectively. For example, a user who attempts to operate a charged particle microscope without attending a proper microscope course lasting several weeks or months is likely to damage the sample or the microscope itself.

[0030] Unfortunately, this need for extensive learning can be significantly amplified by the fact that different types of charged particle microscopes can be operated differently from each other. That is, specialized microscopy training, education, or certification is generally not transferable between different charged particle microscopes. For example, any training that equips someone with the skills to operate a SEM may not equip them with the skills to operate a TEM. Similarly, any training that prepares someone to operate a SEM of type A may not prepare them to operate a SEM of type B. Furthermore, any training that prepares someone to operate a SEM of type A and software version C may not prepare them to operate a SEM of type A and software version D.

[0031] In any case, this enormous operational complexity can prevent users from interacting with charged particle microscopy effectively or intuitively. In other words, this enormous operational complexity can be considered a high barrier to entry in the field of charged particle microscopy.

[0032] Therefore, systems or technologies that can lower this threshold (e.g., make charged particle microscopes more user-friendly or easier to operate) are desirable.

[0033] The various embodiments described herein address this technical problem. One or more embodiments described herein may include systems, computer-implemented methods, apparatuses, or computer program products that facilitate operation of charged particle microscopes using large language models (LLMs). In other words, the various embodiments described herein can utilize large language models (LLMs) (such as ChatGPT) as an accessible, user-friendly interface for charged particle microscopes. Therefore, when implementing the various embodiments described herein, users can interact with or otherwise operate the charged particle microscope by typing or speaking intuitive natural language commands, requests, or queries, without prior knowledge of how to skillfully navigate a large number of complex configurable microscopy settings, buttons, or options. In other words, the various embodiments described herein can be considered to reduce or eliminate the need for users to receive extensive professional education in microscope operation before using or operating the charged particle microscope. Furthermore, the various embodiments described herein can be considered to significantly reduce the learning curve associated with charged particle microscopes or otherwise increase the accessibility or ease of use of charged particle microscopes.

[0034] The inventors have devised various embodiments for achieving this increased accessibility or user-friendliness. As described herein, this increased accessibility or user-friendliness can be achieved by implementing any of the following: LLM monitoring of image or energy spectrum adjustment for user microscopy commands; LLM guidance of image or energy spectrum adjustment in response to user microscopy workflow queries; LLM fault detection of image or energy spectrum adjustment in response to user microscopy fault detection queries; LLM interpretation of image or energy spectrum adjustment in response to user microscopy sample queries; or LLM GUI generation of image or energy spectrum adjustment in response to past user microscopy queries.

[0035] First, consider specific implementations of LLM monitoring of image or energy spectrum adjustments for user microscopy commands. In various implementations, the user of the charged particle microscope can issue commands to the microscope (e.g., commands to perform microscopy tasks, commands to set microscopy parameters to desired values). In various ways, the user can type or speak such commands into the appropriate human-machine interface of the charged particle microscope (e.g., typing commands into text fields, speaking commands into a microphone). In various cases, the charged particle microscope may be loaded with a given sample, and it is possible that the execution of the command will cause unintended or otherwise unusual damage to the given sample. In other words, it is possible that the user may inadvertently damage the given sample due to their lack of experience or expertise in operating the charged particle microscope. The various implementations described herein can help mitigate this damage. Specifically, the various implementations described herein may, in response to the receipt of a command but before the execution or implementation of the command, cause the charged particle microscope to capture an image (e.g., in the case of SEM or TEM) or an energy spectrum (e.g., in the case of EELM) of the given sample. In various respects, such image or energy spectrum acquisition can be facilitated according to any suitable default protocol of charged particle microscopy. In various cases, both the command and the image or energy spectrum can be fed together as a collective input prompt to the LLM, causing the LLM to generate a natural language response, which can be visually played (e.g., on a computer screen) or audibly played (e.g., on an electronic speaker) for the user to see or hear. As described herein, the natural language response can textually describe or explain whether the execution of the command will cause unreasonable or unusual damage to the given sample. In other words, the image or energy spectrum captured by charged particle microscopy can be considered to convey at least some quantity of substantial information about the given sample (e.g., about the optical or chemical properties of the given sample), and the LLM can be considered to evaluate, judge, or otherwise verify the reasonableness or soundness of the user's command based on this substantial information. In some cases, as described herein, this substantive information can be extracted or otherwise enhanced by: identifying documents related to both the command and the image or energy spectrum; obtaining inference task results for the image or energy spectrum from an auxiliary or assistive machine learning model (e.g., predicted or inferred classification labels); or obtaining virtual experimental results regarding both the command and the image or energy spectrum from a digital twin of a charged particle microscope. In practice, such documents, inference task results, or virtual experimental results can be fed to the LLM as supplementary input, providing the LLM with more information about a given sample. In any case, the natural language response can be considered a real-time warning or notification to the user regarding whether the execution of the command will damage the given sample.Therefore, as described in this article, adjusting LLM based on images or energy spectra can help reduce or avoid unintentional or undesirable damage to a given sample, even if the user has not undergone extensive training, education or certification in charged particle microscopy.

[0036] Next, consider specific implementations of LLM guidance for image or energy spectrum adjustment in response to user microscopy workflow queries. In various implementations, a user of a charged particle microscope may expect to perform a specific workflow (e.g., voltage contrast analysis) using the charged particle microscope. However, the user may not know how to correctly perform the specific workflow due to a lack of experience or expertise with charged particle microscopy. Therefore, in various ways, the user may type or speak a question in any suitable human-machine interface of the charged particle microscope, inquiring how to perform the specific workflow. As mentioned above, the charged particle microscope may be loaded with a given sample. In various cases, it is possible that the specific workflow is sample-dependent (e.g., depending on the type of sample to be performed, the specific workflow may consist of different steps or sub-steps). Therefore, similar to the above, the various implementations described herein may, in response to the receipt of a question, cause the charged particle microscope to capture an image or energy spectrum of a given sample via any suitable default microscopy protocol. In various ways, both questions and images or energy spectra can be fed together as collective input cues to the LLM, causing it to generate a natural language response that can be visually or audibly played for the user to see or hear. In various cases, the natural language response can textually describe or explain sample-specific instructions for performing a particular workflow. That is, the natural language response can textually describe or explain in what order the user or charged particle microscope should perform which steps, substeps, or other actions in order to successfully or correctly perform a particular workflow on a given sample. In other words, the images or energy spectra captured by the charged particle microscope can be considered to convey at least some substantial information about a given sample, and the LLM can be considered to utilize this substantial information to identify how best or correctly to perform a particular workflow. As mentioned above, in some cases, this substantial information can be enhanced by relevant documentation, inferred task results, or digital twin simulation results. In any case, the natural language response can be considered to teach or explain to the user how to perform a particular workflow on a given sample. Therefore, as described in this article, adjusting the LLM based on images or energy spectra can help ensure that users are informed in real time how to correctly execute a particular workflow, even if the user has not undergone extensive training, education, or certification in charged particle microscopy.

[0037] Now, consider specific implementations of LLM fault detection in response to a user's microscopy fault detection query, adjusting the image or energy spectrum. In various implementations, a charged particle microscope may be loaded with a given sample, and the charged particle microscope may experience a specific fault (e.g., one or more specified symptoms or error codes) when attempting to operate on the given sample. The user of the charged particle microscope may wish to fault detect, diagnose, or otherwise resolve the specific fault, but may not know how to do so due to a lack of experience or expertise. Therefore, in various ways, the user may type or speak a question into any suitable human-machine interface of the charged particle microscope, where the question may inquire about the specific fault (e.g., what is causing the specific fault, how to correct the specific fault). In various cases, the user's question may describe or otherwise identify at least one known detail of the given sample (e.g., the known composition or identity of the given sample). Various implementations may utilize this known detail about the given sample to determine how to handle the specific fault. In practice, similar to the above, the various implementations described herein may, in response to the receipt of a question, cause the charged particle microscope to capture an image or energy spectrum of the given sample via any suitable default microscopy protocol. In various ways, both questions and images or energy spectra can be fed together as collective input prompts to the LLM, enabling it to generate a natural language response that can be visually or audibly played for the user to see or hear. In various cases, this natural language response can describe or explain in text the cause or solution to a specific malfunction. That is, the natural language response can describe or explain in text what specific settings or components of the charged particle microscope are causing a specific malfunction, or it can describe or explain in text the order in which steps, substeps, or other actions should be performed to successfully or correctly address a specific malfunction. In other words, the images or energy spectra captured by the charged particle microscope can be considered to convey measured information about at least some quantities of a given sample, the questions typed or spoken by the user can be considered to convey at least some known information about a given sample, and the LLM can be considered to compare the measured information with the known information to identify why or how the charged particle microscope is suffering from a specific malfunction. Similar to the above, in some cases, the measured information about a given sample can be enhanced by relevant documentation, inferred task results, or digital twin simulation results. In some respects, the measured data can be further enhanced by obtaining self-diagnostic test results from the charged particle microscope itself. In any case, the natural language response can be interpreted as teaching or explaining to the user why a specific fault is occurring for a given sample, or how to prevent a specific fault from occurring for a given sample.Therefore, adjusting the LLM based on images or energy spectra, as described in this article, can help ensure that users are informed in real time how to properly resolve a particular fault, even if the user has not undergone extensive training, education, or certification in charged particle microscopy.

[0038] Next, consider specific implementations of LLM interpretation in response to a user's microscopy sample query, adjusting the image or energy spectrum. In various implementations, a charged particle microscope may be loaded with a given sample, and the user of the charged particle microscope may wish to determine some specific characteristics or properties of the given sample (e.g., chemical composition, crystal structure, surface roughness). However, the user may not know how to properly utilize the charged particle microscope to determine this specific characteristic or property due to their lack of experience or expertise. Therefore, in various ways, the user may type or speak a question in any suitable human-machine interface of the charged particle microscope, where the question may request identification of the specific characteristic or property. Thus, similar to the above, the various implementations described herein may, in response to the receipt of a question, cause the charged particle microscope to capture an image or energy spectrum of the given sample via any suitable default microscopy protocol. In various ways, both the question and the image or energy spectrum may be fed together as a collective input prompt to the LLM, causing the LLM to generate a natural language response, which may be visually or audibly played for the user to see or hear. In various cases, this natural language response may describe, explain, or otherwise identify the specific characteristic or property of the given sample in textual form. In other words, an image or energy spectrum captured by a charged particle microscope can be considered to convey substantial information about at least some quantity of a given sample, and an LLM can be considered to utilize this substantial information to identify or quantify a specific characteristic or property requested by the user. Similar to the above, in some cases, the substantial information about a given sample can be enhanced by relevant documentation, inference task results, or digital twin simulation results. In any case, this natural language response can be considered to identify a specific characteristic or property of a given sample as requested by the user. Therefore, adjusting an LLM based on images or energy spectra, as described herein, can help inform the user of specific characteristics or properties in real time, even if the user has not undergone extensive training, education, or certification in charged particle microscopy.

[0039] Finally, consider specific implementations of LLM GUI creation in response to past user microscopy queries for image or energy spectrum adjustments. In various implementations, a user can load a given sample into the charged particle microscope. Furthermore, in various aspects, the user can previously inquire about various questions regarding the charged particle microscope or give various commands (e.g., microscopy setup commands, workflow queries, fault detection queries, sample queries). In various cases, the charged particle microscope may have numerous configurable software settings, buttons, knobs, sliders, or options to manage or otherwise control its operation. In various cases, it is possible that various settings in such configurable settings are irrelevant to or unsuitable for a given sample. Furthermore, in various aspects, it is possible that some configurable settings correspond to basic microscopy functions suitable for inexperienced operators, while others correspond to advanced microscopy functions only applicable to experienced or trained operators. Therefore, as mentioned above, the various implementations described herein enable the charged particle microscope to capture images or energy spectra of a given sample via any suitable default microscopy protocol. In various ways, images or energy spectra, along with past queries or commands, can be fed together as collective input prompts to the LLM, enabling the LLM to generate synthetic code that can be compiled, executed, or otherwise run on the charged particle microscope or on any suitable computerized workstation associated with it. As described herein, this synthetic code can define a charged particle microscope GUI tailored to both the given sample and the user. More specifically, the images or energy spectra captured by the charged particle microscope can be considered to convey at least some amount of substantial information about the given sample (which can be enhanced using relevant documentation, inferential mission results, or digital twin simulation results), and the LLM can be considered to utilize this substantial information to identify which of the numerous configurable software settings or options of the charged particle microscope are relevant or irrelevant to the given sample. Therefore, the GUI defined by this synthetic code can mask, hide, or otherwise omit settings or options irrelevant to the sample. Furthermore, a user's past queries or commands can be considered to convey at least some amount of information about how experienced or inexperienced the user is with charged particle microscopy, and the LLM can be considered to use this information to infer which of the numerous configurable software settings or options for charged particle microscopy are suitable or unsuitable for the user (e.g., too advanced for the user). Therefore, the GUI defined by this synthetic code can mask, hide, or otherwise omit settings or options that are unsuitable for the user.Therefore, as described herein, adjusting the LLM based on images or energy spectra and based on past queries or commands can help ensure that the charged particle microscopy GUI presented to the user is commensurate with or appropriate to both the given sample and the inferred level of user experience (e.g., the GUI can vary across samples) (e.g., the GUI can vary based on how much microscopy training the user has undergone).

[0040] Therefore, different samples may require different microscopy settings, protocols, treatments, or preparations, or otherwise be associated with different microscopy settings, protocols, treatments, or preparations, and the various embodiments described herein allow the LLM to take these different microscopy settings, protocols, treatments, or preparations into account when synthesizing responses to user questions or commands. This can be achieved by adjusting the LLM based on sample images or sample energy spectra (e.g., by having the LLM receive sample images or sample energy spectra as input). Such image- or energy-spectrum-adjusted LLMs make charged particle microscopy more accessible or user-friendly (e.g., intuitive to use or operate, regardless of technical or professional background).

[0041] The various embodiments described herein can be considered as computerized tools (e.g., any suitable combination of computer-executable hardware or computer-executable software) that facilitate large language model-assisted operations for charged particle microscopy. In various aspects, such computerized tools may include access components, state components, context components, model components, or renderer components.

[0042] In various embodiments, a charged particle microscope may be present. In various aspects, the charged particle microscope may exhibit any suitable design or configuration (e.g., it may be a SEM, a TEM, an EELM, or a dual-beam microscope). In various cases, the charged particle microscope may include any suitable number of configurable operating settings. In various cases, the configurable operating settings may be any suitable selectively controllable hardware or software characteristics of the charged particle microscope that can be directly adjusted or changed in response to electronic instructions or commands received from the user of the charged particle microscope (e.g., user-controlled voltage or current settings, user-controlled temperature settings, or user-controlled actuator settings). In various aspects, any suitable sample (e.g., a semiconductor wafer or wafer) currently loaded in the charged particle microscope (e.g., currently located or positioned on an actuable stage of the charged particle microscope) may be present.

[0043] In various implementations, an LLM (Layered Learning Model) may exist. In various aspects, the LLM can exhibit any suitable deep learning internal architecture. For example, the LLM may include any suitable number of layers of any suitable type (e.g., input layer, one or more hidden layers, output layer, any of which may be a convolutional layer, dense layer, long short-term memory (LSTM) layer, transformer layer, nonlinear layer, pooling layer, batch normalization layer, or padding layer). As another example, the LLM may include any suitable number of neurons in various layers (e.g., different layers may have the same or different numbers of neurons). Furthermore, the LLM may include any suitable activation function in various neurons (e.g., softmax, sigmoid, hyperbolic tangent, modified linear unit) (e.g., different neurons may have the same or different activation functions). Moreover, the LLM may include any suitable inter-neuron or inter-layer connections (e.g., forward connections, skip connections, recurrent connections).

[0044] Regardless of its specific internal architecture, the LLM can be configured as a generative text-to-text model. That is, the LLM can be configured to receive any suitable text data (which may or may not be accompanied by any suitable numerical data or any suitable graphical data) as input, and the LLM can be configured to produce synthetic text content (e.g., one or more synthetic sentences or sentence fragments) as output that is semantically or substantially based on such input text data (and based on accompanying numerical or graphical data, as appropriate).

[0045] To achieve this, the LLM can be considered to include an encoder section and a synthesizer section. In various respects, the encoder section can be any suitable upstream layer of the LLM configured to receive input text data (and any accompanying numerical or graphical data) and generate embeddings based on that input text data. In various cases, the synthesizer section can be any suitable downstream layer of the LLM configured to receive those embeddings and generate synthesized text content based on those embeddings.

[0046] In various respects, an embedding generated by the encoder portion of an LLM in response to a set of input text, numerical, or graphical data can be considered as any suitable mathematical quantity (e.g., scalar, vector, matrix, tensor, word granularity, or any suitable combination thereof) that numerically represents at least some substantial or semantic aspects of the input text, numerical, or graphical data in a low-dimensional manner. In other words, the embedding may be smaller in size or dimension than such input text, numerical, or graphical data (e.g., in some cases, one or more orders of magnitude smaller); however, despite its small size, the embedding can still be considered to substantially or semantically represent such input text, numerical, or graphical data. In yet another way, the embedding can be considered as a latent vector representation of such input text, numerical, or graphical data.

[0047] In any case, LLM can be expected to increase the accessibility or ease of operation of charged particle microscopy. The computerized tools described herein can achieve this in various ways, depending on the circumstances.

[0048] In some implementations, computerized tools can increase the accessibility or ease of use of charged particle microscopes by utilizing LLM to perform automated sanity or rationality checks on commands received by users of the charged particle microscope.

[0049] In practice, natural language instructions can be associated with charged particle microscopy in various situations. In various aspects, these instructions can be unstructured text or plain text that semantically requests or commands the setting, alteration, or other adjustment of one or more configurable operating settings of the charged particle microscope (e.g., beam voltage settings, beam current settings, stage temperature settings) to one or more desired values ​​or states. In various cases, these instructions can be typed or spoken by the user via any suitable GUI text field or microphone of the charged particle microscope.

[0050] In various implementations, the access component of the computerized tool can electronically access the natural language instruction. For example, the access component can receive, retrieve, or otherwise obtain the natural language instruction from any suitable centralized or decentralized data structure (e.g., graph data structure, relational data structure, hybrid data structure). Similarly, the access component can electronically access an LLM or charged particle microscope. For example, the access component can electronically interact with or communicate with the LLM or charged particle microscope (e.g., send electronic commands to it, read electronic signals from it). In any case, the access component can be considered a conduit through which other components of the computerized tool can electronically interact with the natural language instruction, LLM, or charged particle microscope (e.g., read, write, edit, copy, manipulate, execute, activate, deactivate, or modify the natural language instruction, LLM, or charged particle microscope).

[0051] In various implementations, the state component of the computerized tool can electronically cause the charged particle microscope to scan the sample currently loaded in the charged particle microscope in response to the receipt or access of natural language instructions. In various aspects, such scanning can cause the charged particle microscope to capture images depicting or illustrating at least some portions of the sample (e.g., SEM scan images, TEM scan images), or to capture the energy spectrum of the sample or otherwise associated with the sample (e.g., electron energy loss spectrum). In various cases, the state component can cause the charged particle microscope to perform such scanning according to any suitable default microscopy protocol known or considered non-destructive or non-damaging for a wide range or largely of possible microscopy samples. As a non-limiting example, a default microscopy protocol may involve using a default beam current (e.g., on the order of nanoamperes (nA) or picoamperes (pA)) and a default beam voltage (e.g., less than 5 kilovolts (kV)) that are sufficiently low to not damage, degrade, or otherwise degrade all, most, or any suitable subgroups of any possible samples that the charged particle microscope is expected to or designed to encounter.

[0052] In various implementations, the model component of a computerized tool can electronically generate a natural language response by performing LLM on a natural language instruction, an image or energy spectrum of a sample, and a damage cue associated with the natural language instruction. More specifically, the damage cue can be an interrogation or command that determines whether the execution or implementation of the natural language instruction will damage, corrupt, or otherwise degrade unstructured or plain text of the sample. In various aspects, the model component can cascade the natural language instruction, the image or energy spectrum, and the damage cue. In various cases, the model component can feed this cascade into the input layer of the LLM, which can perform a forward pass through one or more hidden layers of the LLM, and the output layer of the LLM can compute a natural language response based on the activations provided by one or more hidden layers of the LLM.

[0053] In various contexts, a natural language response can be synthetic text that is based on natural language instructions and on images or energy spectra, and substantially or semantically responds to a damage warning. In other words, a natural language response can be unstructured or plain text describing or interpreting whether changing or adjusting one or more configurable operating settings of a charged particle microscope to one or more desired values ​​or states (e.g., requested or commanded by the user) will unreasonably or undesirably damage the sample. Furthermore, images or energy spectra can be considered to inform the LLM of at least some physical, chemical, or compositional information about the sample, and the LLM can use this information to infer or predict whether a user-requested or commanded setting change will damage the sample. Still, the LLM can be considered to utilize images or energy spectra of currently loaded samples to monitor or verify the reasonableness or soundness of natural language instructions, and the natural language response can be considered a conclusion or determination of this monitoring or verification.

[0054] In some cases, by allowing the LLM to consider supplementary or contextual information about an image or energy spectrum, or otherwise derived from an image or energy spectrum, the level of accuracy, completeness, or specificity or detail presented by the natural language response can be increased or otherwise improved. In various implementations, the contextual component of the computerized tool can electronically acquire, collect, or otherwise access such supplementary or contextual information.

[0055] As a non-limiting example, there may exist a set of auxiliary or assistive machine learning models that can be configured to perform corresponding inference tasks based on an input image or input energy spectrum. As some non-limiting examples, any machine learning model in such auxiliary or assistive machine learning models may have been pre-trained to perform: image classification or energy spectrum classification; image segmentation or energy spectrum segmentation; or image regression or energy spectrum regression. Therefore, the context component can execute the appropriate machine learning model from the set of auxiliary or assistive machine learning models on the image or energy spectrum of the currently loaded sample, and such execution can produce multiple inference task results related to the sample (e.g., multiple predicted or inferred classification labels, segmentation masks, or regression outputs). Thus, in various aspects, the model component can cascade these multiple inference task results with natural language instructions, the image or energy spectrum, and a damage warning, and the model component can generate a natural language response by performing LLM on this amplified cascade. In various cases, these multiple inference task results can be considered to provide the LLM with deeper or richer information about the sample, and this deeper or richer information can enable the LLM to make the natural language response more accurate or more detailed.

[0056] As another non-limiting example, a document repository comprising multiple documents may exist. In various cases, each of these multiple documents may be any suitable electronic file (e.g., a Word-doc file, a Portable Document Format (PDF) file, a web page file) that describes, explains, or otherwise indicates, in textual form (or in some cases graphically or numerically), any suitable technical information regarding the physical, chemical, or optical properties of any suitable sample or regarding the design, manufacture, operation, maintenance, or troubleshooting of any suitable charged particle microscope (e.g., the various documents may be maintenance manuals or technical manuals (or parts thereof) for corresponding charged particle microscopes; the various documents may be composition reports or reference tables for corresponding laboratory samples). In various cases, any of these multiple documents may have been written or has been written by a technician or engineer responsible for designing, developing, prototyping, revising, manufacturing, or researching any suitable charged particle microscope or any suitable sample that can be analyzed by a charged particle microscope (e.g., via any suitable word processing software, computer-aided design software, or quantitative analysis software). It should be noted that in some cases, any document may be presented or otherwise have any suitable length or size (e.g., it may be one or several pages in length; it may be tens of pages in length; it may be hundreds of pages in length). In any case, the context component may electronically search the document repository for one or more documents substantially related to the natural language instruction and the image or spectrum. In some aspects, the context component may achieve this via embedding search. For example, the encoder portion of the LLM may be used to generate specific embeddings for the natural language instruction and the image or spectrum; the encoder portion may be used to generate a corresponding embedding for each document in the document repository; and any document whose embedding is closest to or most similar to the specific embedding may be considered related to the natural language instruction and the image or spectrum. Thus, in various aspects, the model component may cascade those related documents with the natural language instruction, the image or spectrum, and the damage prompt, and the model component may generate a natural language response by performing an LLM on this amplified cascade. In various cases, relevant documentation can be considered to provide LLM with deeper or richer information about how the sample is known or expected to interact with charged particle microscopy, and this deeper or richer information can enable LLM to make natural language responses more accurate or detailed.

[0057] As another non-limiting example, a charged particle microscope can be electronically synchronized with a digital twin. In various respects, a digital twin can be any suitable mathematical model or any suitable combination of physics-based models that can numerically, computationally, or analytically predict, forecast, or otherwise simulate how a charged particle microscope, or any suitable part thereof, will respond to any given use case. More specifically, a digital twin can include parameter states, a set of input variables, and a set of output variables. In various respects, the set of input variables can be collectively viewed as operands of the digital twin, the parameter states can be collectively viewed as operators defining the digital twin, and the set of output variables can be computed or calculated by applying the parameter states to the set of input variables mathematically (e.g., via any suitable mathematical function or its components). Thus, the set of input variables can be assigned any numerical value that defines or describes any given use case, and how a charged particle microscope will behave or respond to that given use case (e.g., output variables) can be simulated, predicted, or forecasted by applying the parameter states to the set of input variables. In various cases, any suitable synchronization techniques can be implemented to ensure, or otherwise, that the parameter states of the digital twin closely match the actual physical state of the charged particle microscope (e.g., within any suitable threshold margin of the actual physical state of the charged particle microscope). In various aspects, LLM can generate one or more function calls for the digital twin based on natural language instructions and on images or energy spectra. In various cases, execution of such one or more function calls can cause the digital twin to run or perform one or more virtual experiments with respect to the charged particle microscope and the sample. In various cases, those one or more virtual experiments can produce corresponding simulation results (e.g., corresponding simulated or predicted values ​​of the digital twin's output variables). Therefore, in various aspects, the model component can cascade those simulation results with natural language instructions, images or energy spectra, and damage cues, and the model component can generate a natural language response by performing LLM on this amplified cascade. In various cases, these simulation results can be considered to provide the LLM with deeper or richer information about how the sample is predicted or expected to interact with the charged particle microscope, and this deeper or richer information can enable the LLM to make the natural language response more accurate or detailed.

[0058] In any case, LLM can generate a natural language response that can describe or interpret in textual form whether the natural language instructions provided by the user will or may unintentionally or unintentionally damage or harm the currently loaded sample.

[0059] In various implementations, the presenter component of the computerized tool can electronically present the natural language response to the user in any suitable manner. As a non-limiting example, the presenter component can visually present the natural language response on any suitable computer screen or computer monitor associated with the charged particle microscope, allowing the user to view or read the natural language response. As another non-limiting example, the presenter component can audibly play the natural language response on any suitable speaker associated with the charged particle microscope (e.g., via any suitable text-to-speech technology), allowing the user to hear or perceive the natural language response. In this way, the computerized tool can be considered to warn or notify the user whether a requested or commanded setting change or adjustment will unintentionally damage or harm the sample.

[0060] In some respects, if the LLM determines or concludes that the execution or implementation of a natural language instruction will impair or damage the currently loaded sample, the natural language response may further include one or more recommended values ​​or states that may or should be used in place of those values ​​or states specified in the natural language instruction. In such cases, the renderer component may electronically ignore the natural language instruction (e.g., it may violate the natural language instruction by suppressing the change or adjustment of one or more configurable operating settings to one or more desired values ​​or states). In such cases, the renderer component may alternatively electronically cause the charged particle microscope to change or adjust one or more configurable operating settings to one or more recommended values ​​or states.

[0061] In this way, computerized tools can be considered to increase the accessibility or ease of use of charged particle microscopes by utilizing LLM to automatically prevent users from unintentionally or mistakenly damaging or destroying currently loaded samples. Therefore, even users without experience or familiarity with charged particle microscopes can use or operate them without (or with significantly reduced) concern or risk of unintentional damage to samples.

[0062] In some other implementations, computerized tools can now increase the accessibility or ease of use of charged particle microscopes by leveraging LLM to provide sample-tailored answers to workflow questions posed by users of charged particle microscopes.

[0063] In practice, natural language workflow queries can exist in various contexts related to charged particle microscopy. In various ways, natural language workflow queries can be unstructured text or plain text that semantically requests or commands an explanation of how to correctly or properly perform certain workflows on a charged particle microscope (e.g., what steps are involved in a voltage contrast workflow; what steps are involved in a bending workflow). As described above, natural language workflow queries can be typed or spoken by the user.

[0064] In various implementations, the state component of the computerized tool can electronically enable a charged particle microscope to scan the currently loaded sample via any suitable default microscopy protocol in response to the receipt or access of a natural language workflow query. In various respects, this scanning can enable the charged particle microscope to capture images or energy spectra of the sample.

[0065] In various implementations, the model components of the computerized tool can electronically generate natural language responses by querying natural language workflows and performing LLM on images or energy spectra (e.g., performing LLM on a cascade of natural language workflow queries and images or energy spectra).

[0066] In various cases, the natural language response can be synthesized text based on an image or energy spectrum and substantially or semantically responding to a natural language workflow query. In other words, the natural language response can be unstructured plain text describing or explaining what specific steps, substeps, or sequences of actions the user should perform (as inferred or predicted by the LLM) to execute a particular workflow. Now, some microscopy workflows can be sample-dependent. That is, when performing a given microscopy workflow relative to a first type of sample (e.g., a metal sample), the given microscopy workflow may involve a first sequence of steps or actions, and when performing a given microscopy workflow relative to a second type of sample (e.g., a plastic sample), the given microscopy workflow may alternatively involve a second sequence of steps or actions. For example, a metal sample may require different pre-scanning cleaning or sanitation steps than a plastic sample. As another example, a plastic sample may require a sputtering coating step, while the sputtering coating step may be omitted for a metal sample. Therefore, images or energy dispersive spectroscopy can be considered to inform the LLM of at least some physical, chemical, or compositional information about the currently loaded sample, and the LLM can use this information to infer or predict what specific steps or sequences of actions are required to correctly or appropriately execute a particular workflow on a charged particle microscope for or for the currently loaded sample. Furthermore, in some cases, a particular workflow may simply not be suitable or applicable to the sample (e.g., a particular workflow may be reserved for organic samples, but the LLM may infer that the currently loaded sample is inorganic). In such cases, a natural language response can describe or explain why the particular workflow is not applicable to the currently loaded sample. In any case, the LLM can be considered to utilize images or energy dispersive spectroscopy to provide the user with sample-tailored workflow guidance.

[0067] As described above, by allowing the LLM to consider supplementary or contextual information about the image or energy spectrum, or otherwise derived from the image or energy spectrum, the level of accuracy, completeness, or specificity or detail presented by the natural language response can be increased or otherwise improved. This supplementary or contextual information (e.g., relevant documentation; aiding or assisting in inferring task results; digital twin simulation results) can be acquired, collected, or otherwise accessed electronically by the contextual component. Therefore, any information in this supplementary or contextual information can be fed to the LLM as additional input. As mentioned above, this can be considered as providing the LLM with deeper or richer information about how the charged particle microscope and the currently loaded sample are known, expected, or predicted to interact or relate to each other, and this deeper or richer information allows the LLM to make the natural language response more accurate or detailed.

[0068] In any case, LLM can generate a natural language response that can describe, explain, or teach in textual form how to perform a specific workflow on a charged particle microscope in a manner applicable or suitable for the currently loaded sample.

[0069] In various implementations, as described above, the presenter component of a computerized tool can present natural language responses to the user electronically (e.g., visually or audibly).

[0070] In this way, computerized tools can be considered to increase the accessibility or ease of use of charged particle microscopes by automatically teaching or showing users how to correctly perform the requested workflow for the currently loaded sample through the use of LLM. Therefore, even users without experience or familiarity with charged particle microscopes or the requested workflow can be competent to use or operate the charged particle microscope.

[0071] In further implementations, computerized tools can increase the accessibility or ease of use of charged particle microscopy by leveraging LLM to provide diagnostic or fault detection interpretations of sample conditioning in response to fault questions posed by the user.

[0072] In practice, natural language fault queries can exist in various contexts associated with charged particle microscopy. In various ways, a natural language fault query can be unstructured text or plain text that semantically requests or commands an explanation of why the charged particle microscope is experiencing one or more specific fault symptoms (e.g., why is the charged particle microscope emitting a specific error code; why is the charged particle microscope emitting an alarm or beep; why is the charged particle microscope emitting grinding noise). In some cases, a natural language fault query may include at least some description of the currently loaded sample (e.g., these faults occur when attempting to analyze a steel sample; these faults occur when attempting to analyze a printed circuit board sample). As above, natural language fault queries can be typed or spoken by the user.

[0073] In various implementations, the state component of the computerized tool can electronically enable a charged particle microscope to scan the currently loaded sample via any suitable default microscopy protocol in response to the receipt or access of a natural language fault query. In various respects, this scanning can enable the charged particle microscope to capture an image or energy spectrum of the sample.

[0074] In various implementations, the model components of the computerized tool can electronically generate natural language responses by performing LLM on natural language fault queries and on images or energy spectra (e.g., performing LLM on a cascade of natural language fault queries and images or energy spectra).

[0075] In various cases, the natural language response can be synthetic text based on an image or energy spectrum and substantially or semantically responding to a natural language fault query. In other words, the natural language response can be unstructured or plain text describing or explaining what (e.g., inferred or predicted by the LLM) is causing the charged particle microscope to suffer, reveal, or be subject to one or more specific faults noticed by the user. In some cases, the natural language response may include unstructured or plain text describing or explaining how (e.g., inferred or predicted by the LLM) one or more specific fault symptoms can be resolved or corrected.

[0076] Now, as mentioned above, a natural language fault query can include at least some known information about the currently loaded sample (e.g., at least one physical, chemical, or compositional property that can identify or describe (but briefly) the sample). Furthermore, an image or energy spectrum can be considered to inform the LLM of at least some measured, detected, or estimated information about the currently loaded sample (e.g., measured, detected, or estimated physical, chemical, or compositional properties). In various respects, the LLM can be considered to utilize any differences, inconsistencies, or interrelationships between this known sample information and this measured, detected, or estimated sample information in order to infer or predict why the charged particle microscope is suffering from or exhibiting one or more specific fault symptoms. For example, if the natural language fault query specifies that the currently loaded sample is a silicon wafer sample, the LLM can expect the image or energy spectrum to look or appear in a certain way (e.g., with or without certain visual patterns or artifacts). The degree or manner in which the image or energy spectrum deviates from this expectation can be considered to at least partially indicate or suggest to the LLM what is wrong with the charged particle microscope. Therefore, LLM can be considered as utilizing images or energy spectra to make more evidence-based fault diagnosis of charged particle microscopy, and natural language response can be considered as such diagnosis.

[0077] As described above, by allowing the LLM to consider supplemental or contextual information about the image or energy spectrum of the currently loaded sample, or otherwise derived from the image or energy spectrum of the currently loaded sample, the level of accuracy, completeness, or specificity or detail presented by the natural language response can be increased or otherwise improved. In various embodiments, such supplemental or contextual information (e.g., relevant documentation; aiding or assisting in inferring task results; digital twin simulation results) can be electronically acquired, collected, or otherwise accessed by the contextual component. In some cases, such supplemental or contextual information may include the results of automated self-diagnostic tests or examinations that the charged particle microscope can perform on itself (e.g., in response to function calls generated by the LLM). As mentioned above, such supplemental or contextual information can be considered to provide the LLM with deeper or richer information about how the charged particle microscope and the currently loaded sample are known, expected, or predicted to interact or relate to each other, and this deeper or richer information can enable the LLM to make the natural language response more accurate or detailed.

[0078] In any case, LLM can generate a natural language response that can describe or explain in textual form why the charged particle microscope is experiencing one or more specific malfunction symptoms inquired about by the user.

[0079] In various implementations, as described above, the presenter component of a computerized tool can electronically (e.g., visually or audibly) present natural language responses to the user in any suitable manner.

[0080] In this way, computerized tools can be considered to increase the accessibility or ease of use of charged particle microscopes by automatically teaching or showing users in real time why the charged particle microscope is suffering from a noticed malfunction, or by automatically teaching or showing users in real time how to remedy such a malfunction. Therefore, even users who are inexperienced or unfamiliar with charged particle microscopes can be competent to resolve or handle malfunctions.

[0081] Now, in some further implementations, computerized tools can increase the accessibility or ease of use of charged particle microscopes by leveraging LLM to automatically answer user questions about the currently loaded sample.

[0082] In practice, natural language sample queries can exist in various contexts associated with charged particle microscopy. In various ways, natural language sample queries can be unstructured text or plain text that semantically requests or commands identification of any suitable information about the currently loaded sample (e.g., what the physical or chemical composition of the sample is; what microscopy protocol would be optimal for the sample). As described above, natural language sample queries can be typed or spoken by the user.

[0083] In various implementations, the state component of the computerized tool can electronically enable a charged particle microscope to scan the currently loaded sample via any suitable default microscopy protocol in response to the receipt or access of a natural language sample query. In various respects, this scanning can enable the charged particle microscope to capture an image or energy spectrum of the sample.

[0084] In various implementations, the model components of the computerized tool can electronically generate natural language responses by querying natural language samples and performing LLM on images or energy spectra (e.g., performing LLM on a cascade of natural language sample queries and images or energy spectra).

[0085] In various contexts, a natural language response can be synthesized text based on an image or energy spectrum and substantially or semantically responding to a natural language sample query. For example, if a natural language sample query requests identification of a specific physical or chemical property of a currently loaded sample, the natural language response can textually describe, explain, or identify that specific physical or chemical property (e.g., inferred or predicted by the LLM). Similarly, if a natural language sample query requests identification of the best, optimal, or most preferred microscopy protocol to use for a currently loaded sample, the natural language response can textually describe, explain, or identify that best, optimal, or most preferred microscopy protocol (e.g., inferred or predicted by the LLM). Therefore, an image or energy spectrum can be considered to inform the LLM of at least some physical, chemical, or compositional information about the currently loaded sample, and the LLM can use this information to synthesize a reliable answer to the natural language sample query. That is, the LLM can be considered to utilize an image or energy spectrum to answer sample-related questions posed by a user.

[0086] As described above, by allowing the LLM to consider supplementary or contextual information about the currently loaded sample, or otherwise derived from the image or energy spectrum of the currently loaded sample, the level of accuracy, completeness, or specificity or detail presented by the natural language response can be increased or otherwise improved. This supplementary or contextual information (e.g., relevant documentation; aiding or assisting in inferring task outcomes; digital twin simulation results) can be obtained, collected, or otherwise accessed electronically by the contextual component. As mentioned above, this can be considered as providing the LLM with deeper or richer information about the currently loaded sample, and this deeper or richer information enables the LLM to make the natural language response more accurate or detailed.

[0087] In any case, LLM can generate a natural language response that can answer any questions the user asks about the currently loaded sample in text form.

[0088] In various implementations, as described above, the presenter component of a computerized tool can electronically (e.g., visually or audibly) present natural language responses to the user in any suitable manner.

[0089] In this way, computerized tools can be considered to increase the accessibility or ease of use of charged particle microscopy by automatically answering sample-related questions posed by users in real time using LLM. Therefore, even users without experience or familiarity with charged particle microscopy can be competent to analyze their samples.

[0090] Subsequently, in some further implementations, computerized tools can increase the accessibility or ease of use of charged particle microscopy by leveraging LLM to provide a sample-customized GUI based on past queries provided by the user.

[0091] In practice, multiple past natural language queries or commands that a user has previously queried may exist in various situations. In various aspects, any of these past natural language queries can be unstructured text or plain text requests or commands as described above (e.g., instructing changes or adjustments to configurable operating settings of a charged particle microscope; querying the workflow of a charged particle microscope; querying for faults in a charged particle microscope; querying for currently loaded samples).

[0092] In various implementations, the state component of the computerized tool can electronically command the charged particle microscope, or otherwise cause the charged particle microscope to scan the currently loaded sample via any suitable default microscopy protocol. In various respects, this scanning enables the charged particle microscope to capture an image or energy spectrum of the currently loaded sample.

[0093] In various implementations, the model components of the computerized tool can electronically generate synthetic code by performing LLM on past natural language queries, images or energy spectra of currently loaded samples, and GUI prompts, where the GUI prompts can be unstructured text or plain text that requests or commands to construct user-customized and sample-customized GUIs for charged particle microscopy.

[0094] In various cases, the synthesized code can be one or more lines of computer code written in any suitable computer syntax or decoding language, defining or otherwise serving as a programming script for a custom or customized GUI for charged particle microscopy. Specifically, as mentioned above, charged particle microscopy can have numerous configurable software settings, buttons, knobs, sliders, or options. In some cases, different software settings may be relevant to different types of samples or irrelevant (e.g., protocols or options suitable for organic samples may not be suitable for inorganic samples). Similarly, in some cases, some software settings of charged particle microscopy may be suitable for inexperienced users (e.g., basic or simple settings may be suitable for technicians unfamiliar with charged particle microscopy), while other software settings may be only suitable for experienced users (e.g., advanced or critical settings may be only suitable for technicians who are fully familiar with charged particle microscopy).

[0095] In various cases, images or energy dispersive spectroscopy can be considered to inform the LLM of at least some physical, chemical, or compositional information about the currently loaded sample, and the LLM can use this information to infer or predict which software settings are relevant to the currently loaded sample and which others are not. Furthermore, in some cases, past natural language queries can be considered to indirectly inform the LLM of the user's microscopy experience or training level, and the LLM can use this information to infer or predict which software settings are suitable for the user and which others are not.

[0096] Therefore, the synthesis code created by LLM can define a GUI that includes only those software settings relevant to the currently loaded sample and appropriate for the user's skill level for inference. In other words, the synthesis code created by LLM can define a GUI that omits or excludes any software settings irrelevant to the currently loaded sample or inappropriate for the skill level of the user or technician for inference. In any case, LLM can be considered to utilize images or energy spectra of the currently loaded sample, along with past natural language queries from the user, to provide a customized or tailored GUI for charged particle microscopy.

[0097] As described above, by allowing the LLM to consider supplementary or contextual information about the image or energy spectrum, or otherwise derived from it, the level of accuracy, completeness, or specificity or detail presented by the synthetic code can be increased or otherwise improved. This supplementary or contextual information (e.g., relevant documentation; aiding or assisting in inferring task results; digital twin simulation results) can be obtained, collected, or otherwise accessed electronically by the contextual component. As mentioned above, this can be considered as providing the LLM with deeper or richer information about the currently loaded sample and the user, and this deeper or richer information allows the LLM to make the synthetic code and thus the customized GUI more appropriate or suitable.

[0098] In any case, LLM can generate synthesis code that can define or otherwise serve as the source code for sample customization and user-customized GUIs for charged particle microscopy.

[0099] In various implementations, the presentation component of the computerized tool can electronically compile, run, or otherwise execute synthetic code to visually present the charged particle microscope sample customization and user-customized GUI to the user on any suitable computer screen or monitor.

[0100] In this way, computerized tools can be considered to increase the accessibility or ease of use of charged particle microscopy by automatically generating a GUI for both the current sample and the inferred technique of a user attempting to use charged particle microscopy using an LLM. Therefore, users can operate or interact with charged particle microscopy intuitively or comfortably, regardless of their individual level of microscopy experience.

[0101] The various implementations described herein can be used to solve problems that are inherently technical (e.g., to facilitate large language model assistance for charged particle microscopy operations), non-abstract, and cannot be performed by humans as a set of mental behaviors, using hardware or software. Furthermore, some of the processes performed can be executed by dedicated computers (e.g., electron microscopes such as SEM, TEM, or EELM; artificial neural networks such as LLM; digital twins) to perform specific behaviors relevant to the field of charged particle microscopy.

[0102] For example, such defined actions may include: accessing a natural language instruction provided by a user of a charged particle microscope by a device operatively coupled to the processor, wherein the natural language instruction may request or command a change in the configurable operating settings of the charged particle microscope to a target value; the device, and in response to the receipt of the natural language instruction, causing the charged particle microscope to capture an image or energy spectrum of a sample currently mounted on the stage of the charged particle microscope according to a default microscopy protocol; and the device performing a large language model on both the natural language instruction and the image or energy spectrum of the sample to generate a natural language response that may indicate whether changing the configurable operating settings to the target value will negatively affect the sample.

[0103] For example, actions defined in this way may include: accessing a natural language query provided by a user of the charged particle microscope by a device operatively coupled to the processor, wherein the natural language query may request or command how to perform a microscopy workflow on the charged particle microscope; the device, and in response to the receipt of the natural language query, causing the charged particle microscope to capture an image or energy spectrum of a sample currently mounted on the stage of the charged particle microscope according to a default microscopy protocol; and the device performing a large language model on both the natural language query and the image or energy spectrum of the sample to produce a natural language response that may describe guidance for a microscopy workflow tailored for the sample.

[0104] For example, actions defined in this way may include: a device operatively coupled to the processor accessing a natural language query provided by a user of the charged particle microscope, wherein the natural language query may request or command identification of why the charged particle microscope is experiencing a malfunction regarding a sample currently mounted on the stage of the charged particle microscope; the device, and in response to the receipt of the natural language query, causing the charged particle microscope to capture an image or energy spectrum of the sample according to a default microscopy protocol; and the device performing a large language model on both the natural language query and the image or energy spectrum of the sample to produce a natural language response that explains the cause of the malfunction.

[0105] For example, actions defined in this way may include: a device operatively coupled to the processor accessing a natural language query provided by a user of the charged particle microscope, wherein the natural language query may request or command the identification of properties of a sample currently mounted on the stage of the charged particle microscope; the device, and in response to the receipt of the natural language query, causing the charged particle microscope to capture an image or energy spectrum of the sample according to a default microscopy protocol; and the device performing a large language model on both the natural language query and the image or energy spectrum of the sample to generate a natural language response that may describe or indicate the properties of the sample.

[0106] For example, such defined actions may include: accessing multiple past natural language microscopy queries provided by a user of the charged particle microscope by a device operatively coupled to the processor; the device causing the charged particle microscope to capture an image or energy spectrum of a sample currently mounted on the stage of the charged particle microscope according to a default microscopy protocol; the device performing a large language model on both the multiple past natural language microscopy queries and the image or energy spectrum of the sample to generate synthetic code that defines a graphical user interface for the charged particle microscope tailored to both the sample and the inferred user's microscopy technique level; and the device running the synthetic code to render or activate the graphical user interface.

[0107] In various cases, any action in such a defined action may involve: synthesizing function calls for a digital twin synchronized with a charged particle microscope based on an image or energy spectrum of a sample, via the execution of an LLM; causing the digital twin to perform virtual experiments or simulations based on those synthesized function calls, thereby producing experimental or simulation results about the sample; and feeding such experimental or simulation results as supplementary input to an LLM to help generate any suitable natural language response or synthesized code.

[0108] Actions defined in this way are inherently computerized. In fact, charged particle microscopes (e.g., SEM, TEM, EELM, dual-beam microscopes) are highly technical computerized devices that include specific computerized hardware (e.g., temperature sensors, pressure sensors, voltage sensors, ion beam emitters, electron beam emitters, focusing lenses, ion detectors, electron detectors, beam apertures, fluid valves, actuable sample stages). Without a computer, charged particle microscopes and the operations they perform cannot be realized in any reasonable or feasible way by human thought or by humans using pen and paper. Furthermore, artificial neural networks (e.g., LLMs) are also inherently computerized constructs, including specific software-oriented architectures (e.g., input layers, hidden layers, or output layers, any of which may consist of trainable or non-trainable internal parameters, such as convolutional layers or LSTM layers). Without a computer, artificial neural networks cannot be trained or executed in any reasonable or feasible way by human thought or by humans using pen and paper. Similarly, a digital twin (as the word "digital" in its name suggests) is an inherently computerized or virtual construct used to electronically predict or simulate the future behavior of a charged particle microscope. Without a computer, a digital twin cannot be facilitated or executed by human thought or by humans using pen and paper in any reasonable or feasible way.

[0109] Furthermore, the various embodiments described herein can incorporate various teachings related to the field of charged particle microscopy into practical applications. As mentioned above, charged particle microscopes are highly complex devices that typically require extensive specialized training, education, or certification to learn how to operate. In practice, they can have numerous configurable or selectable settings, buttons, knobs, sliders, or options, which can be overwhelming for inexperienced users. In other words, charged particle microscopes can be considered user-unfriendly, inaccessible, or otherwise difficult to operate.

[0110] The various implementations described herein can help improve this problem by implementing a large language model (LLM) for charged particle microscopy operation. That is, the various implementations described herein can utilize LLM to increase the user-friendliness, accessibility, or ease of operation of charged particle microscopy. As described herein, such increases can be achieved through any of the following: LLM monitoring of image or energy spectrum adjustments for user microscopy commands; LLM guidance for image or energy spectrum adjustments in response to user microscopy workflow queries; LLM fault detection for image or energy spectrum adjustments in response to user microscopy fault detection queries; LLM interpretation of image or energy spectrum adjustments in response to user microscopy sample queries; and LLM GUI creation for image or energy spectrum adjustments in response to past user microscopy queries.

[0111] In practice, when a user of a charged particle microscope issues a microscopy command (e.g., a plain text instruction to adjust or change configurable microscopy settings), various implementations may involve: causing the charged particle microscope to automatically capture an image or energy spectrum of the currently loaded sample; and feeding both the microscopy command and the image or energy spectrum as input to the LLM, causing the LLM to generate a plain text response explaining whether the implementation of the microscopy command will abnormally or undesirably damage the currently loaded sample. That is, the LLM can be considered to use any physical, chemical, or compositional information about the currently loaded sample conveyed by the image or energy spectrum to infer whether the currently loaded sample is likely to be degraded by the microscopy command. It should be noted that such physical, chemical, or compositional information about the currently loaded sample may optionally be enhanced, supplemented, or accompanied by relevant documents found via embedded search, by inference task results predicted by an auxiliary machine learning model, or by simulation results generated by a digital twin. In any case, the various implementations described herein can be considered as providing continuous or intermittent real-time, background monitoring of user commands to alert or notify the user when or whether their commands are inappropriate for the currently loaded sample. This can be considered as increasing the accessibility or user-friendliness of the charged particle microscope, enabling even inexperienced users to operate the microscope without fear of unintentionally or accidentally damaging the sample. Equivalently, this can be considered as protecting or safeguarding the sample from the inexperience of untrained users.

[0112] Furthermore, when a user of a charged particle microscope presents a microscopy workflow question (e.g., a plain text question inquiring how to perform a specific microscopy workflow), various implementations may involve: enabling the charged particle microscope to automatically capture an image or energy spectrum of the currently loaded sample; and feeding both the microscopy workflow question and the image or energy spectrum as input to the LLM, causing the LLM to generate a plain text response that explains or instructs the user on what specific sequence of steps should be performed to correctly or acceptablely apply the specific microscopy workflow to the currently loaded sample. That is, the LLM can be considered as using any physical, chemical, or compositional information about the currently loaded sample conveyed by the image or energy spectrum to infer what order and what microscopy workflow steps should be performed on the currently loaded sample. Similarly, this physical, chemical, or compositional information about the currently loaded sample may optionally be enhanced, supplemented, or accompanied by relevant documentation, auxiliary inference task results, or digital twin simulation results. In any case, the various implementations described herein can be considered as providing the user with real-time, sample-customized guidance or instruction for performing a microscopy workflow. This can be seen as increasing the accessibility or user-friendliness of charged particle microscopes, making it possible for even inexperienced or untrained users to perform workflows using charged particle microscopes.

[0113] Furthermore, when a user of a charged particle microscope provides a microscopy malfunction problem (e.g., a plain text question asking why a particular microscopy malfunction is occurring), various implementations may involve: enabling the charged particle microscope to automatically capture an image or energy spectrum of the currently loaded sample; and feeding both the microscopy malfunction problem and the image or energy spectrum as input to the LLM, causing the LLM to generate a plain text response that explains or describes the possible causes, origins, or solutions for the particular microscopy malfunction. Specifically, the microscopy malfunction problem may identify any suitable known properties of the currently loaded sample, the image or energy spectrum may be considered to convey at least some measured or detected properties of the currently loaded sample, and the LLM may be considered to compare such known properties with the measured / detected properties to infer why the charged particle microscope is not operating as it should or how the charged particle microscope can be repaired or corrected. Similarly, this measured / detected information about the currently loaded sample may optionally be enhanced, supplemented, or accompanied by relevant documentation, auxiliary inference task results, or digital twin simulation results. In any case, the various implementations described herein can be considered as providing users with real-time fault detection recommendations for charged particle microscopes. This can be considered as increasing the accessibility or user-friendliness of charged particle microscopes, enabling even inexperienced or untrained users to handle faults in charged particle microscopes.

[0114] Furthermore, when a user of a charged particle microscope (CPM) presents a microscopic sample problem (e.g., a plain text question requesting identification of some properties of the currently loaded sample), various implementations may involve: causing the CPM to automatically capture an image or energy spectrum of the currently loaded sample; and feeding both the microscopic sample problem and the image or energy spectrum as input to the LLM, causing the LLM to generate a plain text response that identifies or describes the requested properties of the currently loaded sample. That is, the LLM can be considered to infer the requested properties using any physical, chemical, or compositional information about the currently loaded sample conveyed by the image or energy spectrum. Similarly, this physical, chemical, or compositional information about the currently loaded sample may optionally be enhanced, supplemented, or accompanied by relevant documentation, results of auxiliary inference tasks, or results of digital twin simulations. In any case, the various implementations described herein can be considered to provide the user with real-time sample characterization or analysis in response to the user's intuitive natural language question. This can be considered to increase the accessibility or user-friendliness of the CPM, enabling even inexperienced or untrained users to effectively analyze their desired samples using the CPM.

[0115] In some cases, when a history of microscopy problems is provided from a user of a charged particle microscope, various implementations may involve: enabling the charged particle microscope to automatically capture an image or energy spectrum of the currently loaded sample; and feeding both the history of the microscopy problems and the image or energy spectrum as input to an LLM, thereby causing the LLM to generate synthetic code defining a GUI for the charged particle microscope that: presents configurable microscopy settings that are both relevant to the currently loaded sample and suitable for the inferred level of user experience; and hides configurable microscopy settings that are irrelevant to the currently loaded sample or unsuitable for the inferred level of user experience. That is, the LLM can be considered to use any physical, chemical, or compositional information about the currently loaded sample conveyed by the image or energy spectrum to infer which configurable settings of the charged particle microscope are applicable or not applicable to the currently loaded sample. Similarly, the LLM can be considered to use the history of the microscopy problems to infer how much training or skill the user has with the charged particle microscope, and thereby identify which configurable settings of the charged particle microscope are user-operable (e.g., less advanced) or inoperable (e.g., less advanced). In any case, LLM can use this inferred information to build or define a GUI for charged particle microscopy tailored to both the currently loaded sample and the user. This can be considered as increasing the accessibility or user-friendliness of charged particle microscopy, enabling the automatic presentation of a GUI commensurate with the experience and sample expectations of any or all users of charged particle microscopy.

[0116] For at least the reasons stated above, the various embodiments described herein can be considered to solve or improve various problems or shortcomings regarding the accessibility of charged particle microscopy. Therefore, the various embodiments described herein can be considered concrete and practical technical improvements in the field of charged particle microscopy. Consequently, the various embodiments described herein certainly qualify as useful and practical applications of computers.

[0117] Furthermore, the various embodiments described herein can control real-world tangible devices based on the disclosed teachings. For example, the various embodiments described herein can electronically activate, deactivate, or otherwise actuate real-world hardware (e.g., ion beam emitter, ion focusing lens, liquid-carrying valve / pump) of real-world charged particle microscopes (e.g., SEM, TEM, EELM, dual-beam microscopes).

[0118] Figure 1 Example non-limiting block diagrams of scientific instrument module 102 according to various embodiments described herein are shown.

[0119] In various implementations, the scientific instrument module 102 may be implemented by circuitry (such as a programmed computing device) including electrical or optical components. The logical components of the scientific instrument module 102 may be included in a single computing device or, depending on the circumstances, distributed across multiple computing devices communicating with each other. (References herein) Figure 33 and Figure 35 Examples of computing devices that can implement the scientific instrument module 102 individually or in combination are discussed, and references are made to... Figure 34 and Figure 36 Examples of systems or networks of interconnected computing devices that enable the scientific instrument module 102 to be implemented across one or more computing devices in a computer device are discussed.

[0120] Scientific instrument module 102 may include a first logic unit 104, a second logic unit 106, and a third logic unit 108. As used herein, the term "logic unit" may include means for performing a set of operations logically associated. For example, any logic element included in scientific instrument module 102 may be implemented by one or more computing devices programmed with instructions to cause one or more processing devices of the computing device to perform an associated set of operations. In a particular embodiment, a logic element may include one or more non-transitory computer-readable media having instructions that, when executed by one or more processing devices of the one or more computing devices, cause the one or more computing devices to perform an associated set of operations. As used herein, the term "module" may refer to a collection of one or more logic elements that together perform the functions associated with the module. Different logic elements in a module may take the same form or may take different forms. For example, some logic elements in a module may be implemented by a programmed general-purpose processing device, while other logic elements in the module may be implemented by an application-specific integrated circuit (ASIC). As another example, different logic elements in a module may be associated with different sets of instructions executed by one or more processing devices. A module may omit one or more logic elements in the associated diagrams; for example, when the module performs a subset of the operations discussed herein with reference to the module, the module may include a subset of the logic elements depicted in the associated figures.

[0121] In various embodiments, a scientific instrument corresponding to scientific instrument module 102 may be present. In various aspects, the scientific instrument can be any suitable computerized device capable of electronically measuring some scientifically relevant, clinically relevant, or research-related characteristic, property, or attribute of an analytical sample (e.g., a known or unknown mixture, compound, or collection of substances). As a non-limiting example, the scientific instrument can be a scanning electron microscope. In this case, the scientific instrument can measure or determine the surface morphology of the analytical sample. As another non-limiting example, the scientific instrument can be a transmission electron microscope. In this case, the scientific instrument can measure or determine the internal structural details of the analytical sample. As yet another non-limiting example, the scientific instrument can be an electron energy loss microscope. In this case, the scientific instrument can measure or determine the positional count or intensity of the analytical sample within a series of defined energy loss bins or bands. As a more general non-limiting example, the scientific instrument can be any suitable type of charged particle microscope (e.g., some types of microscopes use non-electron ion beams to capture images or energy spectra).

[0122] In various implementations, the first logic 104 can access natural language instructions typed or spoken by a user of the scientific instrument. In various aspects, natural language instructions can be one or more unstructured sentences requesting or commanding the setting of one or more configurable operating parameters of the scientific instrument to one or more target values.

[0123] In various embodiments, the second logic 106 may relate to enabling a scientific instrument to capture an image or energy spectrum of any sample currently loaded in or on the scientific instrument in response to the receipt of a natural language instruction. In various respects, the scientific instrument may utilize any suitable default instrument protocol to capture the image or energy spectrum (e.g., a default beam voltage or beam current known to be non-destructive for all or many different types of samples).

[0124] In various embodiments, third logic 108 may involve performing LLM on both a natural language instruction and an image or energy spectrum of the currently loaded sample. In various cases, such execution may cause the LLM to synthesize a natural language response that describes, interprets, or otherwise indicates whether the fulfillment of the natural language instruction (e.g., whether changing one or more configurable operating parameters to one or more target values) will damage the currently loaded sample. That is, the image or energy spectrum may be considered to convey or contain at least some physical, chemical, or compositional information about the currently loaded sample, and the LLM may be considered to use this information to infer or predict whether the natural language instruction is likely to damage the currently loaded sample. In various cases, third logic 108 may also include presenting the natural language response visually or audibly to a user (e.g., via a computer screen or computer speaker of a scientific instrument).

[0125] Therefore, the scientific instrument module 102 can facilitate the use of a large language model for the operation of charged particle microscopes (e.g., continuously monitoring user-provided commands for the scientific instrument to warn or notify the user whether such commands may damage the loaded sample).

[0126] Figure 2 This is an example non-limiting flowchart of a computer-implemented method 200 according to various embodiments described herein. The operation of the computer-implemented method 200 can be used in any suitable context to perform any suitable operation (e.g., by [the relevant authority]). Figure 1 , Figure 32 , Figure 33 , Figure 34 , Figure 35 and Figure 36 (This refers to the execution or use in conjunction with any of the various modules, computing devices, or graphical user interfaces described). Figure 2 In this context, operations are each instantiated once in a specific order, but they can be reordered or repeated as needed and as appropriate (e.g., different operations can be executed in parallel under suitable circumstances).

[0127] In various aspects, action 202 may include performing a first operation that accesses natural language instructions provided by a user of the charged particle microscope. In various cases, the natural language instructions may request or command the setting of configurable operating parameters of the charged particle microscope to target values. In various cases, the first logic unit 104 may perform or otherwise facilitate action 202.

[0128] In various respects, action 204 may include performing a second operation that, in response to the receipt of a natural language instruction, causes the charged particle microscope to capture an image or energy spectrum of a sample currently mounted on the stage of the charged particle microscope, according to a default microscopy protocol. In various cases, the second logic component 106 may perform or otherwise facilitate action 204.

[0129] In various cases, action 206 may include performing a third operation that applies a large language model to both the natural language instruction and the image or energy spectrum of the sample. This execution may produce a natural language response indicating whether setting the configurable operational parameters to target values ​​will negatively affect the sample.

[0130] Therefore, the computer-implemented method 200 can facilitate the use of large language models for charged particle microscope operation.

[0131] Figure 3 A block diagram illustrating an example non-limiting system that facilitates the operation of charged particle microscopy with the aid of a large language model, according to one or more embodiments described herein.

[0132] In various embodiments, a charged particle microscope 302 may be present. In various aspects, the charged particle microscope 302 may be as described above. That is, the charged particle microscope 302 may be any suitable computerized device that can utilize its constituent hardware (e.g., electron source, anode, condenser lens, condenser aperture, scanning coil, objective lens, objective aperture, deflector, condenser, astigmatism reducer, electron detector, X-ray detector, actuable sample stage) to electronically capture any suitable image or any suitable energy spectrum of any suitable analytical sample. As a non-limiting example, the charged particle microscope 302 may be any suitable SEM. As another non-limiting example, the charged particle microscope 302 may be any suitable TEM. As another non-limiting example, the charged particle microscope 302 may be any suitable scanning transmission electron microscope (STEM). As another non-limiting example, the charged particle microscope 302 may be any suitable EELM. As another non-limiting example, the charged particle microscope 302 may be any suitable dual-beam microscope.

[0133] Although not explicitly shown in the figures, the charged particle microscope 302 can be electronically integrated with any suitable human-machine interface device, either remotely or locally on the charged particle microscope 302. Thus, a user or technician associated with the charged particle microscope 302 can interact with or otherwise control the charged particle microscope 302. Some non-limiting examples of the human-machine interface device could be a keyboard, a keypad, a touchscreen, or a voice command system for the charged particle microscope 302.

[0134] In any case, the charged particle microscope 302 may include a plurality of configurable operating settings 304. In various aspects, each of the plurality of configurable operating settings 304 may be any suitable hardware-related or software-related characteristic of the charged particle microscope 302 that directs, influences, or otherwise instructs how the charged particle microscope 302 operates, functions, or works with respect to any given analytical sample, and may be selectively controlled, altered, adjusted, or otherwise set by a user or technician (e.g., via interaction with a human-machine interface device of the charged particle microscope 302). As a non-limiting example, any of the plurality of configurable operating settings 304 may be a user-controllable voltage setting (e.g., beam voltage) or current setting (e.g., beam current) that allows a user or technician to selectively control the electrodes of the charged particle microscope 302 to selectively increase or decrease the voltage or current within or applied by the charged particle microscope 302. As another non-limiting example, any of the configurable operating settings 304 could be a user-controllable temperature setting that allows a user or technician to control the heater (e.g., stage heater, heating coil) or cooler (e.g., cooling fan, heat pump, refrigerator) of the charged particle microscope 302 to selectively increase or decrease the temperature within or applied by the charged particle microscope 302. As yet another non-limiting example, any of the configurable operating settings 304 could be a user-controllable mechanical actuator setting that allows a user or technician to control the mechanical actuators (e.g., electric motor, sample stage, aperture, fluid pump, or syringe) of the charged particle microscope 302 to selectively move the mechanical actuators. As another non-limiting example, any of the multiple configurable operating settings 304 may be a user-controllable optical device setting that allows a user or technician to control the optical elements (e.g., optical lenses, optical deflectors) of the charged particle microscope 302 in order to selectively change the optical quality (e.g., spot size or position, astigmatism, defocus) applied by the charged particle microscope 302.

[0135] In various respects, the charged particle microscope 302 may have a currently loaded sample 306 or otherwise associated with a currently loaded sample. In various cases, the currently loaded sample 306 may (as its name suggests) be currently loaded on or within the charged particle microscope 302. As a non-limiting example, the currently loaded sample 306 may be currently positioned, located on, or otherwise attached to an actuable stage of the charged particle microscope 302 such that the currently loaded sample 306 can be analyzed or scanned by the charged particle microscope 302. In various cases, the currently loaded sample 306 may be any suitable type of medical, clinical, scientific, or laboratory sample or specimen that exhibits any suitable physical, chemical, compositional, or other properties, attributes, or characteristics. As a non-limiting example, the currently loaded sample 306 may be a sheet taken from a semiconductor substrate or wafer. As another non-limiting example, the currently loaded sample 306 may be a sample or fragment taken from a failed support structure.

[0136] In various implementations, natural language instructions 308 may be present. In various aspects, natural language instructions 308 may be any suitable number of plain text or unstructured sentences or sentence fragments that request (e.g., in the case of an interrogative sentence) or command (e.g., in the case of an imperative sentence) one or more of the multiple configurable operational settings 304 to set one or more target values ​​or states 332 respectively. In various cases, the one or more target values ​​or states 332 may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, one or more categories, or any suitable combination thereof that a user or technician wants or expects to change or adjust the corresponding configurable operational settings of the multiple configurable operational settings 304 to. As some non-limiting examples, natural language instructions 308 may be any of the following: “Set the beam voltage to 30kV”; “Change the beam current to 0.2nA”; or “Can you increase the sample stage temperature by 10%?”.

[0137] It should be understood and appreciated that, unlike explicitly requesting or commanding the change of a specific configurable operating setting among multiple configurable operating settings 304 to one or more target values ​​or states 332, in some cases, natural language instructions 308 may alternatively request or command the charged particle microscope 302 to take, activate, or initiate one or more specified microscopy actions or tasks. As some non-limiting examples, natural language instructions 308 may be any of the following: “Load Inorganic Protocol Suite”; “Start Voltage Comparison Analysis”; or “Please Start Grinding”. For ease of illustration and explanation, the remainder of this disclosure describes various embodiments of natural language instructions 308 including one or more target values ​​or states 332. However, it should be understood that any such embodiments are equally applicable to or extendable to situations where natural language instructions 308 alternatively request or command the initiation or execution of microscopy tasks, actions, or functions without explicitly stating one or more target values ​​or states 332.

[0138] In various situations, natural language instructions 308 may be provided or entered by a user or technician of the charged particle microscope 302 via any suitable human-machine interface device associated with the charged particle microscope 302. As a non-limiting example, the user or technician may type natural language instructions 308 (e.g., via a keyboard, keypad, or touchscreen) into any suitable GUI text field of the charged particle microscope 302, an auxiliary charged particle microscope 302, or a computerized workstation paired with the charged particle microscope. As another non-limiting example, the user or technician may speak aloud into any suitable microphone of the charged particle microscope 302, an auxiliary charged particle microscope 302, or a computerized workstation paired with the charged particle microscope, and any suitable speech-to-text transcription system, service, or technology may convert the user's or technician's spoken words into natural language instructions 308.

[0139] In various implementations, a large language model 310 (hereinafter referred to as "LLM 310") may exist. In various aspects, LLM 310 may include an encoder portion 312 and a synthesizer portion 314. In various cases, the encoder portion 312 may be considered upstream of the synthesizer portion 314. Equivalently, the synthesizer portion 314 may be considered downstream of the encoder portion 312.

[0140] In various respects, the encoder section 312 can exhibit any suitable deep learning internal architecture. In fact, in various cases, the encoder section 312 can have an input layer, one or more hidden layers, and an output layer. In various cases, any of these layers can be coupled together by any suitable inter-neuron or inter-layer connections, such as forward connections, skip connections, or recurrent connections. Furthermore, in various cases, any of these layers can be any suitable type of neural network layer with any suitable learnable or trainable internal parameters. For example, any of such an input layer, one or more hidden layers, or output layer can be a convolutional layer, and the learnable or trainable parameters of the convolutional layer can be a convolutional kernel. As another example, any of such an input layer, one or more hidden layers, or output layer can be a dense layer, and its learnable or trainable parameters can be a weight matrix or a bias value. As yet another example, any of such an input layer, one or more hidden layers, or output layer can be a batch normalization layer, and its learnable or trainable parameters can be a shift factor or a scaling factor. As another example, any of such an input layer, one or more hidden layers, or an output layer can be an LSTM layer, and the learnable or trainable parameters of these LSTM layers can be the input state weight matrix or the hidden state weight matrix. As yet another example, any of such an input layer, one or more hidden layers, or an output layer can be a transformer layer, and the learnable or trainable parameters of these transformer layers can be single-head or multi-head attention blocks or other weight matrices. Furthermore, in various cases, any of such layers can be any suitable type of neural network layer with any suitable fixed or non-trainable internal parameters. For example, any of such an input layer, one or more hidden layers, or an output layer can be a nonlinear layer, a padding layer, a pooling layer, or a cascaded layer.

[0141] Similarly, in various cases, the synthesizer section 314 can exhibit any suitable deep learning internal architecture. In fact, in various cases, the synthesizer section 314 can have an input layer, one or more hidden layers, and an output layer. In various cases, any of these layers can be coupled together by any suitable inter-neuron or inter-layer connections (e.g., forward connections, skip connections, or recurrent connections). Furthermore, in various cases, any of these layers can be any suitable type of neural network layer with any suitable learnable or trainable internal parameters (e.g., any of such an input layer, one or more hidden layers, or output layer can be a convolutional layer, a dense layer, a batch normalization layer, an LSTM layer, or a transformer layer). Furthermore, in various cases, any of such layers can be any suitable type of neural network layer with any suitable fixed or non-trainable internal parameters (e.g., any of such an input layer, one or more hidden layers, or output layer can be a non-linear layer, a padding layer, a pooling layer, or a cascaded layer).

[0142] Regardless of the specific internal architecture implemented within encoder section 312 (e.g., the specific number, type, or organization of layers), encoder section 312 can be configured to receive text data (which may be accompanied by any suitable numerical or graphical data) and generate embeddings based on such input text data. In contrast, regardless of the specific internal architecture implemented within synthesizer section 314, synthesizer section 314 can be configured to receive embeddings generated by encoder section 312 and generate synthesized text content based on such embeddings. As some non-limiting examples, LLM 310 can be any of the following: ChatGPT, Gene.AI, Ollama, Bard, or Claude.

[0143] In various implementations, system 316 may be electronically integrated with charged particle microscope 302, natural language instructions 308, or LLM 310 (e.g., via any suitable wired or wireless electronic connection). As described herein, system 316 may utilize LLM 310 and charged particle microscope 302 to monitor or examine the rationality or soundness of natural language instructions 308.

[0144] In various aspects, system 316 may include processor 318 (e.g., computer processing unit, microprocessor) and non-transitory computer-readable storage 320 operatively or communicatively connected or coupled to processor 318. Non-transitory computer-readable storage 320 may store computer-executable instructions that, when executed by processor 318, cause processor 318 or other components of system 316 (e.g., access component 322, state component 324, context component 326, model component 328, renderer component 330) to perform one or more actions. In various embodiments, non-transitory computer-readable storage 320 may store computer-executable components (e.g., access component 322, state component 324, context component 326, model component 328, renderer component 330), and processor 318 may execute these computer-executable components.

[0145] In various embodiments, system 316 may include access component 322. In various aspects, access component 322 may electronically access charged particle microscope 302 or LLM 310. That is, access component 322 may electronically communicate with or otherwise electronically interact with charged particle microscope 302 or LLM 310 (e.g., transmit electronic instructions or commands to it, receive electronic data from it). Therefore, access component 322 may be considered an agent or channel through which other components of system 316 may interact with, communicate with, or otherwise manipulate charged particle microscope 302 or LLM 310. In various cases, access component 322 may electronically access natural language instructions 308. That is, access component 322 may electronically receive, electronically retrieve, or otherwise electronically obtain natural language instructions 308 from any suitable electronic source or database (e.g., possibly from charged particle microscope 302 or from an associated computerized workstation). In any case, access component 322 can be considered as a proxy or channel through which other components of system 316 can interact with or otherwise manipulate the natural language instruction 308. However, these are merely non-limiting examples. In other cases, access component 322 may be omitted, and any other component of system 316 may communicate or interact directly with charged particle microscope 302, LLM 310, or natural language instruction 308.

[0146] In various embodiments, system 316 may include a status component 324. In various aspects, as described herein, status component 324 may enable charged particle microscope 302 to capture images or energy spectra of the currently loaded sample 306. In some cases, status component 324 may also obtain the current health status of charged particle microscope 302 from a digital twin associated with it.

[0147] In various embodiments, system 316 may include context component 326. In various cases, as described herein, context component 326 may identify various supplementary information that provides valuable or rich context for natural language instructions 308, images or energy spectra of currently loaded sample 306, or the current health status of charged particle microscope 302.

[0148] In various embodiments, system 316 may include model component 328. In various cases, as described herein, model component 328 may perform LLM 310 on natural language instructions 308, on images or energy spectra of the currently loaded sample 306, on the current health status of the charged particle microscope 302, or on any supplementary data obtained through context component 326. In various aspects, such execution may produce a natural language response that interprets or describes whether the fulfillment or implementation of natural language instructions 308 will undesirably harm the currently loaded sample 306.

[0149] In various embodiments, system 316 may include presenter component 330. In various cases, as described herein, presenter component 330 may visually or audibly present natural language responses so that they can be seen or heard by a user or technician of charged particle microscope 302.

[0150] It should be noted that, in various contexts, access component 322, state component 324, context component 326, model component 328, and renderer component 330 can be collectively considered as one or more software components 321 of system 316. In various respects, it should be understood that, for ease of explanation and illustration, one or more software components 321 are generally described herein as comprising five components (e.g., access component 322, state component 324, context component 326, model component 328, and renderer component 330). However, one or more software components 321 are not limited to being implemented as exactly five components in every embodiment. In fact, in some embodiments, the functionality of these five components described herein can be combined in any suitable manner to be implemented in fewer than five components or by fewer than five components (e.g., in some cases, a single component can perform all the functionality described herein with respect to access component 322, state component 324, context component 326, model component 328, and renderer component 330). In other implementations, the functionality of these five components described herein may instead be distributed, separated, split, or segmented in any suitable manner so as to be implemented in or by more than five components (e.g., two or more components may facilitate functionality that can be performed by access component 322; two or more components may facilitate functionality that can be performed by state component 324; two or more components may facilitate functionality that can be performed by context component 326; two or more components may facilitate functionality that can be performed by model component 328; two or more components may facilitate functionality that can be performed by renderer component 330).

[0151] Figure 4A block diagram of an exemplary non-limiting system for facilitating operation of charged particle microscopy with the aid of a large language model, according to one or more embodiments described herein, is illustrated. The system includes a sample image, a sample energy spectrum, and the current microscope health status.

[0152] In various embodiments, the state component 324 may electronically collect various information about the current or present-time state of the charged particle microscope 302 in response to electronic reception or electronic access of natural language command 308. In various aspects, this information may include image 402, energy spectrum 404, or current health status 406. Regarding... Figures 5 to 6 Various non-restrictive aspects are described.

[0153] Figure 5 Example non-limiting block diagrams illustrating how image 402 or energy spectrum 404 can be obtained according to one or more embodiments described herein are shown.

[0154] In various implementations, the state component 324 may, in response to the receipt or access of natural language instructions 308, cause, instruct, or otherwise command the charged particle microscope 302 to electronically scan the currently loaded sample 306. In various respects, such scanning may be performed according to any suitable default microscopy protocol that can be implemented by the charged particle microscope 302 and is known or anticipated to be non-destructive for all, most, many, or any suitable subset of any sample population that the charged particle microscope 302 may potentially encounter in the clinical, scientific, or laboratory fields. In other words, the default microscopy protocol may be any suitable microscopy scan in which any of the plurality of configurable operating settings 304 is set, changed, or adjusted to any suitable default value or state known or anticipated to be safe for a wide range of potential or possible samples. As some non-limiting examples, the default microscopy protocol may involve setting, changing, or adjusting the beam current setting, beam voltage setting, and stage temperature setting of the charged particle microscope 302 to any suitable default ampere, voltage, and temperature values ​​that are known or expected not to damage, harm, or degrade most potential samples. For example, the beam voltage, beam current, and stage temperature of the default microscopy protocol can be set, changed, or adjusted to any low, threshold, or other non-limiting values ​​that are widely recognized as not damaging many samples (e.g., if it is known or expected that most or many different types of samples will not be damaged by a beam voltage below 10 kV, the default microscopy protocol may have a beam voltage setting set to any suitable value below 10 kV; if it is known or expected that most or many different types of samples will not be damaged by a beam current below 0.1 nA, the default microscopy protocol may have a beam current setting set to any suitable value below 0.1 nA; if it is known or expected that most or many different types of samples will not be damaged by a stage temperature of 300 Kelvin (K), the default microscopy protocol may have a stage temperature setting set to 300 K). In some aspects, this scan can produce an image 402. In other aspects, this scan can produce an energy spectrum 404.

[0155] In various cases, image 402 may visually depict or illustrate the currently loaded sample 306 or any portion thereof in any suitable manner. In some cases, image 402 may be an x-by-y pixel array for any suitable positive integers x and y. As another non-limiting example, image 402 may be an x-by-y-by-z voxel array for any suitable positive integers x, y, and z. In various aspects, the visual quality or appearance of image 402 (e.g., brightness, contrast, resolution, color) may vary with or otherwise depend on the default microscopy protocol implemented by charged particle microscope 302 (e.g., it may depend on the default values ​​or states of a plurality of configurable operating settings 304 used by charged particle microscope 302 to scan the currently loaded sample 306). As a non-limiting example, the default microscopy protocol may be any suitable type of backscattered electron detection (BSE) scan. In this case, image 402 may be considered to capture, convey, or otherwise represent various crystallographic, morphological, or magnetic field information about the currently loaded sample 306. As another non-limiting example, the default microscopy protocol can be any suitable type of electron backscatter diffraction (EBSD) scan. In this case, image 402 can be considered to capture, convey, or otherwise represent various crystal structure or orientation information regarding the currently loaded sample 306. As another non-limiting example, the default microscopy protocol can be any suitable cathodoluminescence scan. In this case, image 402 can be considered to capture, convey, or otherwise represent high-resolution morphological information regarding the luminescent portion (if any) of the currently loaded sample 306.

[0156] In various respects, the energy spectrum 404 can be considered a graph of the measured intensity versus energy (or energy loss). As a non-limiting example, the energy spectrum 404 can be any suitable graph whose horizontal axis can represent or span any suitable number of electron energy bars or bands (e.g., measured in eV), and whose vertical axis can represent or indicate the number of electrons detected (e.g., measured in electron count or in arbitrary units (AU)). In other words, the charged particle microscope 302 can irradiate the currently loaded sample 306 with an electron beam (or any other suitable charged particle beam), such irradiated electrons interacting with the currently loaded sample 306 (e.g., passing through the currently loaded sample, bouncing back from the currently loaded sample, emitting X-rays during or after a collision with the currently loaded sample), the unique chemical composition of the currently loaded sample 306 causing such irradiated electrons to interact with the currently loaded sample 306 in different proportions depending on the energy level, and the energy spectrum 404 can be considered to show how many irradiated electrons are detected interacting with the currently loaded sample 306 at each defined energy level. As described above, the visual quality or appearance of the energy spectrum 404 may vary with or otherwise depend on the default microscopy protocol implemented by the charged particle microscope 302. As a non-limiting example, the default microscopy protocol could be any suitable energy-dispersive X-ray spectroscopy (EDS) scan. In this case, the energy spectrum 404 could be considered to capture, convey, or otherwise represent the elemental composition or chemical characterization of the currently loaded sample 306. As another non-limiting example, the default microscopy protocol could be any suitable electron energy loss (EEL) scan. In this case, the energy spectrum 404 could be considered to capture, convey, or otherwise represent the atomic cross-section or atomic concentration of the currently loaded sample 306.

[0157] In some embodiments, state component 324 may cause charged particle microscope 302 to generate only image 402 without generating energy spectrum 404. In other embodiments, state component 324 may cause charged particle microscope 302 to generate only energy spectrum 404 without generating image 402. In other embodiments, state component 324 may cause charged particle microscope 302 to generate both image 402 and energy spectrum 404. In other embodiments, state component 324 may cause charged particle microscope 302 to generate multiple instances of image 402 (e.g., one image generated using default BSE scan, another image generated using default EBSD scan, and yet another image generated using default cathodoluminescence scan) or multiple instances of energy spectrum 404 (e.g., one energy spectrum generated using default EDS scan, and another energy spectrum generated using default EEL scan).

[0158] In any case, image 402 or energy spectrum 404 may be considered to represent, contain or convey at least some information about any suitable physical, chemical or compositional properties, nature or characteristics of the currently loaded sample 306.

[0159] Figure 6 Example non-limiting block diagrams illustrating how the current health status 406 can be obtained according to one or more embodiments described herein are shown.

[0160] In various embodiments, the charged particle microscope 302 can be electronically synchronized with the digital twin 602. In various aspects, the digital twin 602 may include parameter states 604, a set of input variables 606, and a set of output variables 608.

[0161] In various respects, digital twin 602 may be any suitable set or collection of suitable mathematical or physics-based models that can collectively simulate, predict, or otherwise forecast any suitable behavioral details or aspects of charged particle microscope 302. As a non-limiting example, digital twin 602 may include any suitable mass continuity equations, inequalities, or formulas relating in some way to charged particle microscope 302. As another non-limiting example, digital twin 602 may include any suitable energy balance equations, inequalities, or formulas relating in some way to charged particle microscope 302. As another non-limiting example, digital twin 602 may include any suitable heat transfer equations, inequalities, or formulas relating in some way to charged particle microscope 302. As another non-limiting example, digital twin 602 may include any suitable fluid flow equations, inequalities, or formulas relating in some way to charged particle microscope 302. As another non-limiting example, digital twin 602 may include any suitable dynamic or kinematic equations, inequalities, or formulas relating in some way to charged particle microscope 302. As another non-limiting example, digital twin 602 may include any suitable Newtonian or quantum mechanical equations, inequalities, or formulas that are somehow related to charged particle microscope 302. As another non-limiting example, digital twin 602 may include any suitable corrosion or degradation equations, inequalities, or formulas that are somehow related to charged particle microscope 302.

[0162] In any case, digital twin 602 can be considered as a set of mathematical or physical models capable of simulating, predicting, or forecasting something about charged particle microscope 302, and these mathematical or physical models can be considered as consisting of parameter states 604, a set of input variables 606, and a set of output variables 608. Specifically, parameter states 604 can be considered as operators or coefficients defining these mathematical or physical models, the set of input variables 606 can be considered as operands or independent variables of these mathematical or physical models, and the set of output variables 608 can be considered as simulation, prediction, or forecast results calculated by these mathematical or physical models.

[0163] In various respects, for any suitable positive integer s, the parameter state 604 of the digital twin 602 may include s parameters: parameter 604(1) to parameter 604(s). In various cases, each parameter of parameter state 604 may be any suitable mathematical quantity that can represent a corresponding physical or theoretical characteristic, property, or property of the charged particle microscope 302. For example, parameter 604(1) may be a scalar, vector, matrix, tensor, or any suitable combination thereof that can represent a first physical or theoretical characteristic, property, or property of the charged particle microscope 302. Similarly, parameter 604(s) may be a scalar, vector, matrix, tensor, or any suitable combination thereof that can represent a s-th physical or theoretical characteristic, property, or property of the charged particle microscope 302. As some non-limiting examples, any parameter of parameter state 604 may represent: the length of the charged particle microscope 302 or any part or component thereof; the width of the charged particle microscope 302 or any part or component thereof; the height of the charged particle microscope 302 or any part or component thereof; the thickness of the charged particle microscope 302 or any part or component thereof; the radius of curvature of the charged particle microscope 302 or any part or component thereof; the mass or density of the charged particle microscope 302 or any part or component thereof; the stiffness of the charged particle microscope 302 or any part or component thereof; the damping coefficient ... The resistance of any part or component of the charged particle microscope 302; the electrical impedance of any part or component of the charged particle microscope 302; the thermal resistance of any part or component of the charged particle microscope 302; the thermal conductivity of any part or component of the charged particle microscope 302; the thermal capacity of any part or component of the charged particle microscope 302; the optical opacity of any part or component of the charged particle microscope 302; the optical aberration coefficients (e.g., defocus coefficient, 2x astigmatism coefficient) of any part or component of the charged particle microscope 302; the decoherence time of any part or component of the charged particle microscope 302; or the quantum Hamiltonian element of any part or component of the charged particle microscope 302.

[0164] In some cases, any parameter of parameter state 604 may represent a physical or theoretical characteristic, property, or property that is expected to be non-transient, constant, or fixed. That is, the value of such a physical or theoretical characteristic, property, or property is expected not to drift, decay, or otherwise change over time or with the use of the charged particle microscope 302. However, in other cases, any parameter of parameter state 604 may represent a physical or theoretical characteristic, property, or property that is expected to be transient, non-fixed, or non-constant. That is, the value of such a physical or theoretical characteristic, property, or property is expected to drift, decay, or otherwise change slowly or rapidly over time or with the use of the charged particle microscope 302.

[0165] It should be noted that, in various respects, it may be possible that any parameter of parameter state 604 cannot be directly or explicitly controlled or selected by any of the multiple configurable operating settings 304. Despite the lack of such direct and explicit control, it is possible that any parameter of parameter state 604 can be indirectly influenced or changed by one or more of the multiple configurable operating settings 304. As a non-limiting example, assume that parameter state 604 includes aberration coefficient parameters. While aberration coefficients can be considered useful theoretical properties that help to quantitatively describe the optical performance or behavior of the charged particle microscope 302, the charged particle microscope 302 may lack aberration coefficient knobs, sliders, joysticks, physical buttons, or software buttons that allow explicit and direct selection of specific desired aberration coefficient values. Nevertheless, the configurable knobs, sliders, joysticks, physical buttons, or software buttons (e.g., defocus knob, astigmatism corrector knob, stage actuator joystick, voltage knob, temperature knob) that the charged particle microscope 302 does have can still indirectly affect the aberration coefficients of the charged particle microscope 302 (e.g., adjusting any one of the defocus knob, astigmatism corrector knob, stage actuator joystick, voltage knob, or temperature knob can increase or decrease the aberration coefficients of the charged particle microscope 302).

[0166] In various respects, for any suitable positive integer t, the set of input variables 606 of the digital twin 602 may include t variables: input variable 606(1) to input variable 606(t). In various cases, each input variable in the set of input variables 606 may be any suitable mathematical quantity that can represent any suitable dimension, feature, detail, or aspect of a use case that the charged particle microscope 302 may encounter or experience. For example, input variable 606(1) may be a scalar, vector, matrix, tensor, or any suitable combination thereof that can represent the first dimension, feature, detail, or aspect of a use case that the charged particle microscope 302 may encounter. Similarly, input variable 606(t) may be a scalar, vector, matrix, tensor, or any suitable combination thereof that can represent the t-th dimension, feature, detail, or aspect of a use case that the charged particle microscope 302 may encounter. As some non-limiting examples, any input variable in the set of input variables 606 may represent: the mass or density of the analyte sample that can be analyzed by the charged particle microscope 302; the chemical composition of the analyte sample that can be analyzed by the charged particle microscope 302; the crystal structure of the analyte sample that can be analyzed by the charged particle microscope 302; the absorption coefficient of the analyte sample that can be analyzed by the charged particle microscope 302; the preheating time provided or allocated to the charged particle microscope 302 for the analysis of the sample; the running time provided or allocated to the charged particle microscope 302 for the analysis of the sample; the cooling time provided or allocated to the charged particle microscope 302 for the analysis of the sample; the maximum or minimum voltage level to be used by the charged particle microscope 302 for the analysis of the sample; the maximum or minimum radiation level to be used by the charged particle microscope 302 for the analysis of the sample; the maximum or minimum fluid flow rate to be used by the charged particle microscope 302 for the analysis of the sample; or the maximum or minimum temperature level to be used by the charged particle microscope 302 for the analysis of the sample.

[0167] In various respects, for any suitable positive integer u, the set 608 of output variables of the digital twin 602 may include u variables: output variable 608(1) to output variable 608(u). In various cases, each output variable in the set 608 may be any suitable mathematical quantity that can represent any suitable characteristic, property, nature, feature, detail, or behavior of the charged particle microscope 302 that is expected to be simulated, predicted, or predicted. For example, output variable 608(1) may be a scalar, vector, matrix, tensor, or any suitable combination thereof that can represent a first simulated, predicted, or predicted characteristic, property, nature, feature, detail, or behavior of the charged particle microscope 302. Similarly, output variable 608(u) may be a scalar, vector, matrix, tensor, or any suitable combination thereof that can represent a uth simulated, predicted, or predicted characteristic, property, nature, feature, detail, or behavior of the charged particle microscope 302. As a non-limiting example, any output variable in the set of output variables 608 may represent a simulated image or simulated energy spectrum that can be captured or generated by the charged particle microscope 302 in response to any given use scenario. As another non-limiting example, any output variable in the set of output variables 608 may represent the total or marginal amount of degradation or wear that the charged particle microscope 302 or any part or component thereof may experience or accumulate in response to any given use scenario. As yet another non-limiting example, any output variable in the set of output variables 608 may represent the total or marginal amount of fuel, calibrator, or electricity that can be consumed by the charged particle microscope 302 during or in response to any given use scenario. As yet still another non-limiting example, any output variable in the set of output variables 608 may represent the amount of remaining or remaining service life of the charged particle microscope 302 in response to any given use scenario (e.g., expressed in hours, runs, or scans).

[0168] Therefore, the digital twin 602 can simulate, predict, or forecast how the charged particle microscope 302 will respond to any particular use case. Specifically, the set of input variables 606 can be assigned any suitable values ​​corresponding to or otherwise defining that particular use case, parameter states 604 can be applied to the set of input variables 606 (e.g., according to any mathematical function, operation, or formula that makes up the digital twin 602), and the set of output variables 608 can be equal to the computational result of such application (e.g., product, difference, sum, quotient).

[0169] In various respects, the status component 324 may command or otherwise cause the digital twin 602 to return or provide a current health status 406 in response to electronic reception or access of natural language instructions 308. In various cases, the current health status 406 may be any suitable electronic data calculated or tracked by the digital twin 602 and relating to the current health or maintenance of the charged particle microscope 302. As a non-limiting example, the current health status 406 may include: a recent or current version of parameter status 604 or any subset thereof (e.g., may include currently or recently synchronized aberration coefficients of the charged particle microscope 302); or a recently calculated or recorded version of the output variable set 608 or any subset thereof (e.g., may include the current-time total wear or cumulative wear of the charged particle microscope 302 or any component thereof; may include the current-time remaining service life of the charged particle microscope 302 or any component thereof).

[0170] Figure 7 A block diagram of an example non-limiting system for facilitating the operation of charged particle microscopy with the assistance of a large language model, according to one or more embodiments described herein, is illustrated. The system includes a set of relevant documents, a set of inference task results, and a set of simulation results.

[0171] In various implementations, the context component 326 may electronically collect, or otherwise provide, various supplemental or additional information about the context of the enhanced natural language instruction 308, image 402, energy spectrum 404, or current health status 406. In various aspects, such supplemental or additional information may include a relevant document set 702, an inference task results set 704, or a simulation results set 706. Regarding... Figures 8 to 12 Various non-restrictive aspects are described.

[0172] Figures 8 to 12 Example non-limiting block diagrams illustrating how a relevant document set 702, an inference task result set 704, or a simulation result set 706 can be obtained according to one or more embodiments described herein.

[0173] First, consider Figure 8 In various respects, context component 326 can electronically execute LLM 310 on natural language instructions 308, on images 402, on energy spectra 404, on current health status 406, or on any suitable combination thereof. In various cases, such execution may cause LLM 310 to produce some synthetic text 804.

[0174] More specifically, context component 326 may cascade natural language instructions 308, image 402, energy spectrum 404, and current health state 406 together. It should be noted that in some cases, such cascading may include any other combination of the foregoing items (e.g., may include less than all of natural language instructions 308, image 402, energy spectrum 404, and current health state 406). In various cases, context component 326 may feed or route this cascade to the input layer of encoder section 312. In various cases, this cascade may complete a forward pass through one or more hidden layers of encoder section 312. In various aspects, the output layer of encoder section 312 may compute or otherwise compute embedding 802 based on activation maps or feature maps provided by one or more hidden layers of encoder section 312.

[0175] In various cases, embedding 802 can be considered a latent vector representation that encoder part 312 believes or infers corresponds to a cascade of natural language instruction 308, image 402, energy spectrum 404, or current health state 406. More specifically, embedding 802 can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, or any suitable combination thereof. In various aspects, the dimension of embedding 802 (e.g., the total number or cardinality of numerical elements within embedding 802) can be smaller (e.g., in some cases, many orders of magnitude smaller) than the total or cumulative dimension of natural language instruction 308, image 402, energy spectrum 404, or current health state 406. In various cases, despite its smaller dimension, embedding 802 can still be considered to represent at least some substantial or semantic content of natural language instruction 308, image 402, energy spectrum 404, or current health state 406 (albeit in a hidden or non-obvious manner). In other words, embedding 802 can be considered a compact or compressed numerical representation of natural language instruction 308, image 402, energy spectrum 404, or current health state 406. It should be noted that embedding 802 can be considered to represent natural language instruction 308, image 402, energy spectrum 404, or current health state 406 in a potentially, fuzzy, or otherwise hidden manner, because a third party without a connection or relationship to encoder section 312 will not be able to recreate or infer natural language instruction 308, image 402, energy spectrum 404, or current health state 406 solely from embedding 802.

[0176] Now, in various ways, the embedding 802 can be fed or routed to the input layer of the synthesizer section 314. In various ways, the image 802 can be passed forward through one or more hidden layers of the synthesizer section 314. In various ways, the output layer of the synthesizer section 314 can compute or otherwise compute the synthesized text 804 based on activation maps or feature maps provided by one or more hidden layers of the synthesizer section 314.

[0177] In various respects, the synthesized text 804 can be one or more declarative sentences or sentence fragments generated by the synthesizer section 314 based on the embedding 802. It should be noted that the synthesized text 804 is not an estimate or approximate reconstruction of the natural language instruction 308, image 402, energy spectrum 404, or current health state 406. Rather, the synthesized text 804 can be any suitable number of synthesized sentences that are semantically or substantially related to the embedding 802 and therefore to the natural language instruction 308, image 402, energy spectrum 404, or current health state 406. In some cases, the synthesized text 804 can be considered to contain illusions that are semantically or substantially related to the natural language instruction 308, image 402, energy spectrum 404, or current health state 406.

[0178] In various ways, context component 326 may ignore, discard, or delete the synthesized text 804. However, context component 326 may record, save, store, or otherwise maintain the embedding 802. In other words, context component 326 may extract the embedding 802 from encoder portion 312 (e.g., from the hidden layer of LLM 310).

[0179] Now, consider Figure 9In various embodiments, a document repository 902 may exist. In various aspects, the document repository 902 may include a plurality of documents 904. In various cases, for any suitable positive integer n>1, the plurality of documents 904 may include n documents: document 904(1) to document 904(n). In various cases, each of the plurality of documents 904 may be any suitable electronic file (e.g., a word-doc file, a PDF file, a web page file) that describes, teaches, illustrates, indicates, or otherwise conveys, in textual (or in some cases graphical or numerical) one or more technical features, details, or aspects of any suitable charged particle microscope (which may or may not include charged particle microscope 302), or any suitable microscopic sample (which may or may not include the currently loaded sample 306). As some non-limiting examples, any of the multiple documents in 904 may be a maintenance manual, repair manual, schematic diagram, failure mode report, or any part thereof, which describe or explain: the technical or scientific design of any suitable charged particle microscope (e.g., listing or illustrating the different components, parts, or subsystems of various microscopes and describing how these components, parts, or subsystems work); how to operate any suitable charged particle microscope (e.g., listing or illustrating the different user-configurable settings or controls of various microscopes and explaining what such settings or controls do); any suitable information regarding the intended use or intended operation of any suitable charged particle microscope (e.g., showing or explaining whether various microscopes are designed to handle or not designed to handle). This includes: various operating conditions or usage scenarios for processing; any suitable information regarding how to maintain any suitable charged particle microscope (e.g., listing, illustrating, or explaining the various maintenance tasks expected to be performed for various microscopes); any suitable information regarding fault detection for any suitable charged particle microscope (e.g., describing or explaining how to resolve various fault symptoms for various microscopes); or any suitable information regarding the material properties or characteristics shown in a list of any suitable samples that are routinely or commonly analyzed by the microscope (e.g., listing, describing, or explaining how different types of samples should be analyzed or how different types of samples should be prepared for analysis; listing, describing, or explaining how different types of samples can be affected by different types of microscopes or different microscopy settings).

[0180] It should be recognized and understood that any document in a Multiple Documents 904 can be any subsection or sub-section of a larger document. For example, any document in a Multiple Documents 904 can be a chapter, section, paragraph, or even sentence from a longer document.

[0181] In various respects, context component 326 can electronically generate multiple embeddings 906 by performing LLM 310 as described above on each of the multiple documents 904.

[0182] As a non-limiting example, context component 326 may perform LLM 310 on document 904(1), and context component 326 may extract embedding 906(1) from LLM 310 during the execution. More specifically, context component 326 may feed or route document 904(1) to the input layer of encoder section 312, document 904(1) may complete forward passes through one or more hidden layers of encoder section 312, and the output layer of encoder section 312 may compute or otherwise compute embedding 906(1) based on activation maps or feature maps provided by one or more hidden layers of encoder section 312. It should be noted that embedding 906(1) may have the same format, size, or dimension as embedding 802 (e.g., embedding may be a vector representing a sentence with uniform dimension or uniform size; sentence-by-sentence embeddings of a paragraph may be aggregated or averaged together to produce the embedding of that paragraph; section-by-section embeddings of a chapter may be aggregated or averaged together to produce the embedding of that section or chapter; section-by-section or chapter-by-chapter embeddings of the entire document may be aggregated or averaged together to produce the embedding of the entire document), and therefore embedding 906(1) may be considered as a potential vector representation of document 904(1). In various cases, embedding 906(1) may then be completed via forward pass through synthesizer section 314, but context component 326 may ignore, disregard, or remove any synthesized text content created by synthesizer section 314 based on embedding 906(1).

[0183] As another non-limiting example, context component 326 may perform LLM 310 on document 904(n), and context component 326 may extract embedding 906(n) from LLM 310 during this execution. In practice, as described above, context component 326 may feed or route document 904(n) to the input layer of encoder section 312, document 904(n) may complete a forward pass through one or more hidden layers of encoder section 312, and the output layer of encoder section 312 may compute or otherwise compute embedding 906(n) based on activation maps or feature maps provided by one or more hidden layers of encoder section 312. Therefore, embedding 906(n) may have the same format, size, or dimensions as embedding 802, and thus embedding 906(n) may be considered a latent vector representation of document 904(n). As above, embedding 906(n) can then complete the forward pass through synthesizer section 314, but context component 326 can ignore, disregard, or delete any synthesized text content created by synthesizer section 314 based on embedding 906(n).

[0184] In various cases, embeddings 906(1) to 906(n) can be collectively considered to form multiple embeddings 906.

[0185] In various respects, context component 326 can electronically determine or identify a set of relevant documents 702 by comparing embedding 802 with a plurality of embeddings 906. Specifically, for each given embedding among the plurality of embeddings 906, context component 326 can calculate any suitable error or similarity value between that given embedding and embedding 802. As some non-limiting examples, such error or similarity value may involve: mean absolute error (MAE) calculation; mean squared error (MSE) calculation; cosine similarity calculation; Euclidean distance calculation; or cross-entropy calculation. In any case, context component 326 can determine that the set of relevant documents 702 is any document among the plurality of documents 904 whose embedding (e.g., in 906) is most similar to or closest to embedding 802. As a non-limiting example, for any appropriate positive integer m, context component 326 can identify m documents among the plurality of documents 904 that have embeddings most similar to or closest to embedding 802, and such m documents can be considered the set of relevant documents 702. That is, the relevant document set 702 may include m documents: relevant document 702(1) to relevant document 702(m). In other words, relevant document 702(1) may be any document among multiple documents 904 whose embedding is closest to or most similar to embedding 802, while relevant document 702(m) may be any document among multiple documents 904 whose embedding is the m-th closest to or the m-th most similar to embedding 802.

[0186] In any case, the relevant document set 702 may be considered to be substantially or semantically related in some way to the natural language instruction 308, image 402, energy spectrum 404, or current health state 406. As a non-limiting example, suppose the natural language instruction 308 requests or commands that the beam voltage and stage temperature of the charged particle microscope be set to certain desired values. In this case, any relevant document in the relevant document set 702 may be structured or unstructured text describing or explaining: how the beam voltage setting or stage temperature setting of the charged particle microscope 302 can be manipulated or adjusted; how, given the current health state 406, manipulation or adjustment of the beam voltage setting or stage temperature setting is known or expected to affect the charged particle microscope 302; or, given the current health state 406, how samples whose captured images or energy spectra match or resemble image 402 or energy spectrum 404 are known or expected to respond to changes in beam voltage or stage temperature. Therefore, the relevant document set 702 can be considered to provide valuable contextual information about natural language instructions 308, images 402, energy spectra 404, or current health status 406.

[0187] Next, consider Figure 10 In various implementations, a set of available deep learning models 1002 may exist. In various aspects, for any suitable positive integer p, the set of available deep learning models 1002 may include p models: available deep learning model 1002(1) to available deep learning model 1002(p). In various cases, any available deep learning model in the set of available deep learning models 1002 may exhibit any suitable deep learning internal architecture. In fact, in various cases, any available deep learning model in the set of available deep learning models 1002 may have an input layer, one or more hidden layers, and an output layer. In various cases, any such layer may be coupled together by any suitable inter-neuronal or inter-layer connections (e.g., forward connections, skip connections, or recurrent connections). Furthermore, in various cases, any such layer may be any suitable type of neural network layer with any suitable learnable or trainable internal parameters (e.g., any of such an input layer, one or more hidden layers, or output layer may be a convolutional layer, a dense layer, a batch normalization layer, an LSTM layer, or a transformer layer). Furthermore, in various cases, any of such layers can be any suitable type of neural network layer with any suitable fixed or untrainable internal parameters (e.g., any of such input layers, one or more hidden layers, or output layers can be non-linear layers, padding layers, pooling layers, or cascaded layers).

[0188] Regardless of its specific internal architecture, each available deep learning model in the set of 1002 can be configured to perform a corresponding inference task on an input sample image or an input sample energy spectrum. In various respects, the inference task can be any suitable computerized prediction or forecast relating to any appropriate detail of any sample whose image or energy spectrum is being analyzed. As a non-limiting example, any available deep learning model in the set of 1002 can be configured to perform classification on an input sample image or an input sample energy spectrum to classify a given sample into one of two or more defined classes or categories (e.g., chemical composition category, failure mode category, structure presence category, structure absence category). As another non-limiting example, any available deep learning model in the set of 1002 can be configured to perform segmentation on an input sample image or an input energy spectrum to classify each discrete portion of such an input image or energy spectrum (e.g., individual pixels / voxels or groups thereof, individual energy boxes or groups thereof) into one of two or more defined classes or categories. As another non-limiting example, any available deep learning model in the set of 1002 can be configured to perform regression on an input sample image or an input energy spectrum to compute one or more continuously varying scalars, vectors, matrices, or tensors for each of such input images or energy spectra (e.g., regression may include resolution enhancement, denoising, bounding box computation, or any other suitable data transformation). It should be understood that different available deep learning models in the set of 1002 can be configured to perform different inference tasks on the input image or input energy spectrum. Furthermore, it should be understood that any available deep learning model in the set of 1002 can be configured to receive image 402 as input alone, any available deep learning model in the set of 1002 can be configured to receive energy spectrum 404 as input alone, or any available deep learning model in the set of 1002 can be configured to receive both image 402 and energy spectrum 404 as input.

[0189] In various respects, the context component 326 can electronically generate an inference task result set 704 by executing the available deep learning model set 1002 on the image 402 or on the energy spectrum 404, respectively.

[0190] As a non-limiting example, context component 326 may execute the available deep learning model 1002(1) on image 402 or on energy spectrum 404, and such execution may produce an inference task result 704(1). More specifically, context component 326 may feed image 402 or energy spectrum 404 (each individually, or a cascade of both) into the input layer of the available deep learning model 1002(1), such input data may perform forward passes through one or more hidden layers of the available deep learning model 1002(1), and the output layer of the available deep learning model 1002(1) may compute or otherwise compute the inference task result 704(1) based on activation maps or feature maps provided by one or more hidden layers of the available deep learning model 1002(1). Thus, the inference task result 704 may be any classification label, segmentation mask, or regression output that the available deep learning model 1002(1) has predicted or inferred for image 402 or for energy spectrum 404.

[0191] As another non-limiting example, context component 326 may execute the available deep learning model 1002(p) on image 402 or on energy spectrum 404, and such execution may produce an inference task result 704(p). More specifically, context component 326 may feed image 402 or energy spectrum 404 (each individually, or a cascade of both) into the input layer of the available deep learning model 1002(p), such input data may perform forward propagation through one or more hidden layers of the available deep learning model 1002(p), and the output layer of the available deep learning model 1002(p) may compute or otherwise compute the inference task result 704(p) based on activation maps or feature maps provided by one or more hidden layers of the available deep learning model 1002(p). Thus, the inference task result 704 may be any classification label, segmentation mask, or regression output that the available deep learning model 1002(p) has predicted or inferred for image 402 or for energy spectrum 404.

[0192] In all cases, inference task results 704(1) to inference task results 704(p) can be collectively considered as the set of inference task results 704.

[0193] Therefore, the inference task result set 704 can be considered to provide valuable supplementary information about image 402 or energy spectrum 404 and thus about the currently loaded sample 306.

[0194] Now, consider Figure 11In various implementations, a digital twin prompt 1102 may exist. In various aspects, the digital twin prompt 1102 may be one or more unstructured or plain text sentences or sentence fragments that request or command the digital twin 602 to run or execute virtual experiments regarding the natural language instruction 308. As a non-limiting example, for any suitable positive integer q, the digital twin prompt 1102 may be the following sentence: “The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by current health state 406. q different experiments are designed for the digital twin 602 to test how the sample or microscope will react when following the natural language instruction 308.”

[0195] Therefore, in various cases, context component 326 can electronically execute LLM 310 on natural language instructions 308, image 402, energy spectrum 404, current health status 406, digital twin cue 1102, or any suitable combination thereof. In various cases, such execution can cause LLM 310 to produce a digital twin function call set 1104. More specifically, context component 326 can cascade natural language instructions 308, image 402, energy spectrum 404, current health status 406, and digital twin cue 1102 together. It should be noted that in some cases, this cascade can include any other combination of the foregoing items (e.g., it can include less than all of natural language instructions 308, image 402, energy spectrum 404, current health status 406, and digital twin cue 1102). In various aspects, context component 326 can feed this cascade to the input layer of encoder section 312. In various aspects, this cascade can complete the forward pass through one or more hidden layers of encoder section 312. In various cases, the output layer of encoder section 312 may compute or otherwise compute one or more embeddings (not shown) based on activation maps or feature maps provided by one or more hidden layers of encoder section 312. In various cases, the one or more embeddings may be routed to the input layer of synthesizer section 314. In various aspects, the one or more embeddings may perform forward passes through one or more hidden layers of synthesizer section 314, and the output layer of synthesizer section 314 may compute or otherwise compute the digital twin function call set 1104 based on activation maps or feature maps provided by one or more hidden layers of synthesizer section 314.

[0196] In various cases, as mentioned above, digital twin prompt 1102 may request or command the creation of q different virtual experiments by digital twin 602. Therefore, digital twin function call set 1104 may include q calls: digital twin function call 1104(1) to digital twin function call 1104(q). In various cases, each digital twin function call in digital twin function call set 1104 may be considered as one or more corresponding synthetic computer code lines that can be read by digital twin 602 and define (e.g., in any syntax understood by the digital twin, such as JSON code or Python code) a corresponding virtual experiment that can be executed by digital twin 602. As a non-limiting example, digital twin function call 1104(1) may be one or more first synthetic computer code lines that define a first virtual experiment that can be run on digital twin 602. In other words, digital twin function call 1104(1) can specify that LLM310 infers that the input variable set 606 or parameter state 604 should be assigned to investigate the first value of natural language instruction 308. As another non-limiting example, digital twin function call 1104(q) can be one or more q-th synthetic computer code lines, the definition of which can run on the q-th virtual experiment of digital twin 602. That is, digital twin function call 1104(q) can specify that LLM310 infers that the input variable set 606 or parameter state 604 should be undertaken or assigned to the input variable set or parameter state to investigate the q-th value of natural language instruction 308.

[0197] Next, consider Figure 12 In various implementations, context component 326 may electronically instruct, command, or otherwise cause digital twin 602 to run, execute, or execute each digital twin function call in digital twin function call set 1104, thereby producing simulation result set 706.

[0198] As a non-limiting example, context component 326 may enable digital twin 602 to execute digital twin function call 1104(1). That is, context component 326 may enable digital twin 602 to assign any value specified in digital twin function call 1104(1) to input variable set 606 or parameter state 604, and digital twin 602 may accordingly run a first virtual experiment using those newly assigned values. In various respects, such first virtual experiment may produce simulation result 706(1), wherein simulation result 706(1) may be any output variable in output variable set 608 that digital twin 602 has computed for or as a result of digital twin function call 1104(1). It should be noted that in some cases, simulation result 706(1) may be considered to indicate what health state the charged particle microscope 302 will have when the natural language instruction 308 is implemented according to any first use case defined or specified by digital twin function call 1104(1).

[0199] As another non-limiting example, context component 326 enables digital twin 602 to execute digital twin function call 1104(q). That is, context component 326 enables digital twin 602 to assign any value specified in digital twin function call 1104(q) to input variable set 606 or parameter state 604, and digital twin 602 can accordingly run the q-th virtual experiment using those newly assigned values. In various aspects, such q-th virtual experiment can produce simulation result 706(q), where simulation result 706(q) can be any output variable in output variable set 608 that digital twin 602 has computed for or as a result of digital twin function call 1104(q). As above, in some cases, simulation result 706(q) can be considered to indicate what health state the charged particle microscope 302 will have when the natural language instruction 308 is implemented according to any q use case defined or specified by digital twin function call 1104(q).

[0200] In all cases, simulation results 706(1) to simulation results 706(q) can be collectively considered as the set of simulation results 706.

[0201] Therefore, the simulation results set 706 can be considered to provide valuable supplementary information about the charged particle microscope 302, natural language commands 308, images 402, or energy spectrum 404.

[0202] Figure 13 A block diagram of an example non-limiting system for facilitating operation of charged particle microscopy with the aid of a large language model, according to one or more embodiments described herein, is illustrated, the system including natural language responses and synthesized code.

[0203] In various implementations, model component 328 may utilize LLM 310 to electronically generate natural language response 1302 or synthetic code 1304 based on natural language instructions 308 and any information or data collected or obtained by state component 324 or context component 326. About Figure 14 Various non-restrictive aspects are described.

[0204] like Figure 14 As shown, a damage prompt 1401 may exist. In various aspects, the damage prompt 1401 may be one or more unstructured or plain text sentences or sentence fragments that request or command an assessment of the sample damage probability of the natural language instruction 308. In other words, the damage prompt 1401 may inquire whether the execution of the natural language instruction 308 will or may result in damage to the currently loaded sample 306. As a non-limiting example, the damage prompt 1401 may be the following sentence: “The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by its current health status 406. Will executing the natural language instruction 308 on the microscope degrade the sample?” As another non-limiting example, the damage prompt 1401 may be the following sentence: “Determine whether following the natural language instruction 308 will damage the sample shown in image 402 or energy spectrum 404.”

[0205] Therefore, under various conditions, model component 328 can electronically execute LLM 310 on natural language instructions 308, images 402, energy spectra 404, current health status 406, relevant document sets 702, inference task result sets 704, simulation result sets 706, damage alerts 1401, or any suitable combination thereof. Under various conditions, such execution can cause LLM 310 to generate natural language responses 1302 or synthetic codes 1304. More specifically, context component 326 can cascade natural language instructions 308, images 402, energy spectra 404, current health status 406, relevant document sets 702, inference task result sets 704, simulation result sets 706, and damage alerts 1401 together. As described above, it should be noted that, in various cases, the cascade may include any other combination of the foregoing items (e.g., it may include less than all of the natural language instructions 308, image 402, energy spectrum 404, current health status 406, relevant document set 702, inference task result set 704, simulation result set 706, and damage indication 1401). In various cases, model component 328 may feed the cascade to the input layer of encoder section 312. In various aspects, the cascade may perform forward propagation through one or more hidden layers of encoder section 312. In various cases, the output layer of encoder section 312 may compute or otherwise compute one or more embeddings (not shown) based on activation maps or feature maps provided by one or more hidden layers of encoder section 312. In various cases, the one or more embeddings may be routed to the input layer of synthesizer section 314. In various respects, the one or more embeddings may complete the forward pass through one or more hidden layers of the synthesizer section 314, and the output layer of the synthesizer section 314 may compute or otherwise compute the natural language response 1302 or the synthesized code 1304 based on the activation map or feature map provided by one or more hidden layers of the synthesizer section 314.

[0206] In various respects, as shown in the figure, the natural language response 1302 may include a sample corruption explanation 1402. In various cases, the sample corruption explanation 1402 may be one or more unstructured or plain text statements or sentence fragments that semantically answer the corruption prompt 1401. That is, the sample corruption explanation 1402 may be synthetic text that describes or states whether following, fulfilling, or otherwise implementing the natural language instruction 308 (as requested by the user or technician of the charged particle microscope 302) will degrade, damage, worsen, or otherwise corrupt the currently loaded sample 306.

[0207] As a non-limiting example, suppose that natural language instruction 308 requests or commands that the beam voltage of charged particle microscope 302 be set to 50 kV, and suppose (unknown to the user or technician) that the currently loaded sample 306 has a chemical composition that would be negatively affected or damaged by exposure to the 50 kV beam. As mentioned above, image 402 or energy spectrum 404 may be considered to convey at least some information about the chemical, physical, or compositional properties of the currently loaded sample 306. Because LLM 310 may be adjusted based on image 402 or energy spectrum 404 (e.g., may receive the image or energy spectrum as input), LLM 310 may be considered to have access to this chemical, physical, or compositional information. Therefore, LLM 310 may infer or predict that natural language instruction 308 may damage the currently loaded sample 306, and LLM 310 may accordingly synthesize a sample damage explanation 1402 to state or describe that such damage would occur if natural language instruction 308 were followed. It should be noted that the relevant document set 702, the inference task result set 704, or the simulation result set 706 may be considered as additional, supplementary, contextual, or otherwise enriched information that assists or helps the LLM 310 to correctly or accurately synthesize the sample corruption interpretation 1402 (e.g., correctly or accurately inferring or predicting that the natural language instruction 308 will corrupt the currently loaded sample 306).

[0208] As another non-limiting example, suppose natural language instruction 308 requests or commands that the stage temperature of charged particle microscope 302 be set to 400 Kelvin, and suppose (unknown to the user or technician) the currently loaded sample 306 has a chemical composition that will be negatively affected or damaged by exposure to such elevated temperature. Similarly, image 402 or energy spectrum 404 may be considered to convey at least some information about the chemical, physical, or compositional properties of the currently loaded sample 306. Because LLM 310 can be adjusted based on image 402 or energy spectrum 404, LLM 310 can infer or predict that natural language instruction 308 may damage the currently loaded sample 306, and can accordingly synthesize a sample damage interpretation 1402 to state or describe that damage will occur. As above, the relevant document set 702, the inference task results set 704, or the simulation results set 706 may be considered as additional, supplementary, contextual, or otherwise enriching information that assists or helps LLM 310 in correctly or accurately synthesizing the sample damage interpretation 1402.

[0209] In some cases, the damage warning 1401 may not only ask the natural language instruction 308 whether it will damage the currently loaded sample 306, but also whether it will damage the charged particle microscope 302 itself. As a non-limiting example, the damage warning 1401 may include the following sentence: “The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by its current health status 406. Will the natural language instruction 308 deteriorate or damage the sample or the microscope?” In such cases, the natural language response 1302 may include a microscope damage explanation 1404. In various cases, the microscope damage explanation 1404 may be one or more unstructured or plain text statements or sentence fragments that semantically answer the damage warning 1401 regarding the charged particle microscope 302. That is, the microscope damage explanation 1404 may be synthetic text describing or stating whether following, fulfilling, or otherwise implementing the natural language instruction 308 (as requested by a user or technician) will deteriorate, damage, worsen, or otherwise harm the charged particle microscope 302 itself.

[0210] Consider again the above non-limiting example, where natural language instruction 308 requests or commands that the beam voltage of charged particle microscope 302 be set to 50 kV. Now, suppose (unbeknownst to the user or technician) the charged particle microscope 302 has a nearly depleted cathode ray tube, and suppose that attempting 50 kV with the cathode ray tube nearly depleted could cause excessive wear or damage to the charged particle microscope 302. In various cases, the current health state 406 can be considered to convey (directly or indirectly) the near depletion of the cathode ray tube of the charged particle microscope 302. Because LLM 310 can be adjusted based on the current health state 406, LLM 310 can be considered to have access to that information regarding the near-depleted cathode ray tube. Therefore, LLM 310 can infer or predict that natural language instruction 308 may damage or harm the charged particle microscope 302, and LLM 310 can accordingly synthesize microscope damage explanation 1404 to state or describe that such damage will occur. It should be noted that the relevant document set 702 or simulation results set 706 may be considered as additional, supplementary, contextual, or otherwise enriched information that assists or helps the LLM 310 in correctly or accurately synthesizing the microscope damage interpretation 1404 (e.g., correctly or accurately inferring or predicting that the natural language instruction 308 will damage the charged particle microscope 302).

[0211] As another non-limiting example, suppose natural language instruction 308 requests or commands the vacuum chamber of charged particle microscope 302 to be set to 0.0001 Pascals, and suppose (unknown to the user or technician) the currently loaded sample 306 has a chemical or physical composition that would potentially damage the charged particle microscope 302 by bursting under such pressure. As mentioned above, image 402 or energy spectrum 404 can be considered to convey at least some information about the chemical, physical, or compositional properties of the currently loaded sample 306. Because LLM 310 can be adjusted based on image 402 or energy spectrum 404, LLM 310 can be considered to have access to this chemical, physical, or compositional information, and therefore it can be inferred or predicted that the currently loaded sample 306 may burst due to natural language instruction 308 and thereby damage the charged particle microscope 302, and a microscope damage explanation 1404 can be synthesized accordingly to state or describe that such damage may occur. Similarly, the relevant document set 702, the inference task result set 704, or the simulation result set 706 can be considered as additional, supplementary, contextual, or otherwise enriching information that assists or helps the LLM 310 in correctly or accurately synthesizing the microscope damage interpretation 1404.

[0212] In various respects, as mentioned above, natural language instruction 308 may request or command one or more configurable operational settings of a plurality of configurable operational settings 304 to be set, changed, or adjusted to one or more target values ​​or states 332. In cases where LLM 310 infers or predicts that following natural language instruction 308 will harm the currently loaded sample 306 or will harm the charged particle microscope 302, natural language response 1302 may include one or more recommended values ​​or states 1406 in various respects. In various cases, one or more recommended values ​​or states 1406 may be considered alternative versions of one or more target values ​​or states 332, which LLM 310 has inferred or predicted will not harm or damage the current sample 306 or the charged particle microscope 302.

[0213] As a non-limiting example, again assume that natural language instruction 308 requests or commands a beam voltage of 50 kV. Furthermore, assume that the LLM 310 has inferred or predicted (based on image 402 or energy spectrum 404, as supplemented by current health status 406, relevant document set 702, inference task results set 704, simulation results set 706, or any combination thereof) that a 50 kV beam voltage will damage the currently loaded sample 306 or charged particle microscope 302. In this case, the LLM 310 can infer or predict (based on 402 or 404, as supplemented by 406, 702, 704, 706, or any combination thereof) that such damage can be avoided with a beam current of 15 kV (e.g., here, 15 kV can be considered as one or more recommended values ​​or status 1406). Therefore, the LLM310 can synthesize a natural language response 1302 to state, describe, or explain that any sample damage or microscope damage caused by a 50kV beam voltage can be avoided by using a 15kV beam voltage instead.

[0214] In some implementations, as shown in the figure, the natural language response 1302 may include a skipped step explanation 1408. In various cases, the skipped step explanation 1408 may be one or more unstructured or plain text sentences or sentence fragments that describe, state, or otherwise identify one or more steps, actions, or tasks that a user or technician may perform with respect to the charged particle microscope 302 or the currently loaded sample 306 to avoid any damage inferred or predicted to be caused by the natural language instruction 308. That is, in some cases, the LLM 310 may infer or predict that the natural language instruction 308 will be non-destructive for the currently loaded sample 306 or for the charged particle microscope 302, but only after performing another step, action, or task that the user or technician has not yet performed (e.g., the user or technician unintentionally skipped or forgot).

[0215] As a non-limiting example, suppose natural language instruction 308 requests or commands charged particle microscope 302 to perform voltage contrast analysis, and suppose (unbeknownst to the user or technician) that the currently loaded sample 306 should first be cleaned (e.g., with an alcohol solution) before the voltage contrast analysis. As mentioned above, image 402 or energy spectrum 404 can be considered to convey at least some information about the chemical, physical, or compositional properties of the currently loaded sample 306. Because LLM 310 can be adjusted based on image 402 or energy spectrum 404, LLM 310 can infer or predict that the currently loaded sample 306 is contaminated or has not been cleaned, and that cleaning is necessary before the voltage contrast analysis. Therefore, LLM 310 allows the skipped step interpretation 1408 to state or describe such inferences in textual form. Similarly, the relevant document set 702, the inference task result set 704, or the simulation result set 706 can be considered as additional, supplementary, contextual, or otherwise enriched information that helps or assists the LLM 310 in correctly or accurately synthesizing the skipped step interpretation 1408 (e.g., correctly or accurately inferring or predicting that the currently loaded sample 306 has not been cleaned and that such cleaning is required before starting voltage comparison analysis).

[0216] In various respects, as mentioned above, the LLM 310 can generate synthetic code 1304 in addition to (or in some cases, in place of) the natural language response 1302. In various cases, the synthetic code 1304 can be one or more lines of computer code (e.g., Python code, C++ code, JSON code) that can (at execution, runtime, or compilation time) perform any suitable computerized action concerning the charged particle microscope 302 or the natural language response 1302. As a non-limiting example, in a case where the natural language response 1302 includes one or more recommended values ​​or states 1406, the execution of the synthetic code 1304 can cause the charged particle microscope 302 to automatically set, change, or adjust one or more of the multiple configurable operational settings 304 to one or more recommended values ​​or states 1406 instead of one or more target values ​​or states 332.

[0217] In various embodiments, the presenter component 330 may electronically perform any suitable action regarding the natural language response 1302 or the synthetic code 1304. As a non-limiting example, the presenter component 330 may visually present the natural language response 1302 on any suitable electronic display (e.g., a computer screen) of the charged particle microscope 302 or any suitable computerized workstation associated with the charged particle microscope 302. In this case, the natural language response 1302 may be seen or perceived by a user or technician of the charged particle microscope 302. As another non-limiting example, the presenter component 330 may audibly play the natural language response 1302 on any suitable electronic speaker of the charged particle microscope 302 or any suitable computerized workstation associated with the charged particle microscope 302. In this case, the natural language response 1302 may be heard or perceived by a user or technician of the charged particle microscope 302. In either case, the presenter component 330 may be considered as a warning or notification to a user or technician of unintentional or undesirable harm that may occur in the performance or following of natural language instructions. Therefore, even if the user or technician is inexperienced or untrained in the field of charged particle microscopy, they can still operate the charged particle microscope 302 without much concern about accidentally damaging the currently loaded sample 306 or the charged particle microscope 302 itself. In this way, the system 316 is considered to help protect the currently loaded sample 306 or the charged particle microscope 302 itself from the user's or technician's lack of experience.

[0218] In some cases, the presenter component 330 may electronically execute, run, or compile synthetic code 1304. In various cases, this allows the charged particle microscope 302 to automatically implement one or more recommended values ​​or states 1406 instead of one or more target values ​​or states 332. In other words, microscopy settings requested by the user or technician can be ignored, and microscopy settings inferred by the LLM 310 for the currently loaded sample 306 can be automatically implemented. Similarly, the system 316 is intended to help protect the currently loaded sample 306 or the charged particle microscope 302 itself from the lack of experience of the user or technician.

[0219] In some implementations, a more general outcome prompt may replace the damage prompt 1401. That is, instead of specifically asking whether the implementation of the natural language instruction 308 will damage the currently loaded sample 306 or charged particle microscope 302, the outcome prompt may instead ask how the currently loaded sample 306 or charged particle microscope 302 will respond to the implementation of the natural language instruction 308. In other words, the outcome prompt may be one or more unstructured or plain text sentences or sentence fragments that request or command any consequences (good or bad, positive or negative) that may occur regarding the currently loaded sample 306 or charged particle microscope 302 in accordance with the natural language instruction 308. As a non-limiting example, the outcome prompt may be the following sentence: “The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by current health status 406. How will the implementation of natural language instruction 308 affect the sample or the microscope?” As another non-limiting example, the result prompt could be the following sentence: "The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by current health status 406. Tell me what will happen to the sample or microscope if the natural language instruction 308 is followed."

[0220] In this context, sample damage interpretation 1402 can be replaced by a more general sample result interpretation, and similarly, microscope damage interpretation 1404 can be replaced by a more general microscope result interpretation. In various respects, the sample result interpretation can be one or more synthetic statements or sentence fragments describing or stating how the currently loaded sample 306 will be affected when natural language instruction 308 is followed, performed, or otherwise implemented. Similarly, the microscope result interpretation can be one or more synthetic statements or sentence fragments describing or stating how the charged particle microscope 302 will be affected when natural language instruction 308 is followed, performed, or otherwise implemented.

[0221] As a non-limiting example, suppose that natural language instruction 308 requests or commands that the beam voltage of charged particle microscope 302 be set to 50 kV, and suppose (unknown to the user or technician) that the currently loaded sample 306 has a chemical composition that will not be damaged or degraded by exposure to a 50 kV beam, but will be non-uniformly charged by exposure to a 50 kV beam (resulting in undesirable charging artifacts). As mentioned above, image 402 or energy spectrum 404 can be considered to convey at least some information about the chemical, physical, or compositional properties of the currently loaded sample 306. Because LLM 310 can be adjusted based on image 402 or energy spectrum 404 (e.g., it can receive the image or energy spectrum as input), LLM 310 can be considered to have access to this chemical, physical, or compositional information. Therefore, LLM 310 can infer or predict that the currently loaded sample 306 may be non-uniformly charged, and LLM 310 can accordingly synthesize sample outcome interpretations to state or describe that such charging will occur if the natural language instruction 308 is followed. As above, the relevant document set 702, the inference task outcome set 704, or the simulation outcome set 706 can be considered as additional, supplementary, contextual, or otherwise enriching information that assists or helps LLM 310 in correctly or accurately synthesizing sample outcome interpretations (e.g., correctly or accurately inferring or predicting that the currently loaded sample 306 will be non-uniformly charged due to the natural language instruction 308).

[0222] In this way, the various embodiments described herein can be considered to proactively predict the possible consequences that the implementation of natural language instructions 308 will have on the currently loaded sample 306 or charged particle microscope 302, even if such consequences may not be damage or degradation. Similarly, system 316 can be considered to help protect users or technicians from their own lack of microscopy experience in real time.

[0223] Figures 15 to 16 Example non-limiting flowcharts illustrating computer-implemented methods according to one or more embodiments described herein are provided.

[0224] First, consider Figure 15 In various embodiments, action 1502 may include a natural language request (e.g., 308) by a device (e.g., via 322) operatively coupled to a processor (e.g., 318) to set controllable operating parameters (e.g., one of 304) of the charged particle microscope (e.g., 302) to a desired value (e.g., one of 332).

[0225] In various respects, action 1504 may include, by the device (e.g., via 324) and in response to a natural language request, causing the charged particle microscope to capture an image (e.g., 402) or energy spectrum (e.g., 404) of the currently loaded sample (e.g., 306) using a default microscopy protocol.

[0226] In various cases, action 1506 may include accessing the current health status of the charged particle microscope (e.g., 406) by the device (e.g., via 324) in response to a natural language request and from a digital twin (e.g., 602) synchronized with the charged particle microscope.

[0227] In various cases, action 1508 may include identifying one or more documents (e.g., 702) related to a natural language request, image or energy spectrum, or current health status by the device (e.g., via 326) and via an embedded search of a document database (e.g., 902).

[0228] In various respects, action 1510 may include performing one or more available deep learning models (e.g., 1002) on an image or energy spectrum by the device (e.g., via 326) to produce one or more inference task results (e.g., 704). In various cases, as shown, computer-implemented method 1500 may proceed to action 1602 of computer-implemented method 1600.

[0229] Now, consider Figure 16 In various implementations, action 1602 may include the device (e.g., via 326) operating the digital twin based on one or more function calls (e.g., 1104) generated by a large language model (e.g., 310) based on natural language requests, images or energy spectra, or current health status. In various cases, this may cause the digital twin to produce one or more simulation results (e.g., 706).

[0230] In various respects, action 1604 may include the execution of a large language model by the device (e.g., via 328) on a natural language request, on an image or energy spectrum, on a current health status, on one or more documents, on one or more inference task results, or on one or more simulation results. In various cases, this may cause the large language model to produce a natural language response (e.g., including 1302 of 1402) indicating whether setting controllable operating parameters to desired values ​​would harm the currently loaded sample.

[0231] In various cases, action 1606 may include a natural language response visually or audibly presented by the device (e.g., via 330) on an electronic display or electronic speaker associated with the charged particle microscope. In this way, untrained or inexperienced users of the charged particle microscope may be alerted or notified in real time whether their requests may inadvertently or unintentionally damage the currently loaded sample.

[0232] Thus far, various embodiments of the natural language instruction 308 have been described. That is, various embodiments have been described in which a user or technician requests or commands the charged particle microscope 302 to adjust its settings in a specific manner or otherwise perform certain microscopy tasks. However, the user or technician may have various other types of requests to the charged particle microscope 302. Figures 17 to 30 This describes non-restrictive examples of various other types of requests.

[0233] First, consider Figure 17 Instead of providing natural language instructions 308, users or technicians of the charged particle microscope 302 can alternatively provide natural language workflow queries 1702.

[0234] In various respects, the natural language workflow query 1702 can be any suitable number of plain text or unstructured sentences or sentence fragments, whose request or command identifies how to properly perform, conduct, or otherwise implement any suitable microscopy workflow on or using the charged particle microscope 302. As some non-limiting examples, the natural language workflow query 1702 can be any of the following: “Explain how to perform voltage contrast analysis on this microscope”; “What steps are required for bending analysis?”; or “How should I perform diffraction analysis?”.

[0235] As described above, the user or technician of the charged particle microscope 302 may provide or input the natural language workflow query 1702 via any suitable human-machine interface device associated with the charged particle microscope 302 (e.g., the user or technician may type or speak the natural language workflow query 1702).

[0236] As above, the status component 324 may electronically acquire, receive, retrieve, or otherwise access the image 402, energy spectrum 404, or current health status 406 in response to the receipt or access of the natural language workflow query 1702.

[0237] As above, the context component 326 can electronically acquire, receive, retrieve, or otherwise access the set of inference task results 704 by executing the set of available deep learning models 1002 on the image 402 or the energy spectrum 404, respectively.

[0238] Now, in various situations, context component 326 may electronically identify the relevant document set 702 based on natural language workflow query 1702 instead of natural language instruction 308. As a non-limiting example, context component 326 may cascade natural language workflow query 1702, image 402, energy spectrum 404, current health status 406, or any combination thereof, and may perform LLM 310 on this cascade. Such execution may cause LLM 310 to produce a specific embedding that substantially or semantically represents natural language workflow query 1702, image 402, energy spectrum 404, or current health status 406 (e.g., regarding...). Figure 8 (As described). Therefore, the context component 326 can perform an embedded search via the document repository 902 to identify the relevant document set 702. In this case, the relevant document set 702 can be considered to be substantially or semantically related to or relevant to the natural language workflow query 1702, rather than to the natural language instruction 308. In other words, it can be as about Figures 8 to 9 The relevant document set 702 is obtained as described, but the natural language instruction 308 is replaced by the natural language workflow query 1702.

[0239] Similarly, in various cases, context component 326 may electronically obtain simulation result set 706 based on natural language workflow query 1702 instead of natural language instruction 308. As a non-limiting example, digital twin prompt 1102 may be one or more unstructured or plain text sentences or sentence fragments requesting or commanding digital twin 602 to run or execute virtual experiments regarding natural language workflow query 1702. For example, digital twin prompt 1102 may be the following sentence: “The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by current health status 406. q different experiments are designed for digital twin 602 to test how the sample or microscope will respond to the microscopy workflow identified in natural language workflow query 1702.” Therefore, context component 326 may cascade natural language workflow query 1702, image 402, energy spectrum 404, current health status 406, digital twin prompt 1102, or any combination thereof, and may perform LLM 310 on this cascade. This execution allows the LLM 310 to generate a set of digital twin function calls related to natural language workflow queries 1702, images 402, energy spectra 404, or current health status 406 (e.g., regarding...). Figure 11(As described). Therefore, context component 326 enables digital twin 602 to run the set of function calls, thereby producing a set of simulation results 706. In this case, the set of simulation results 706 can be considered to have been generated from a virtual experiment of any microscopic workflow specified in the natural language workflow query 1702, rather than from a virtual experiment of the natural language instruction 308. In other words, as about Figures 11 to 12 The simulation result set 706 is obtained as described, but the natural language instruction 308 is replaced by the natural language workflow query 1702.

[0240] Therefore, in various respects, model component 328 can electronically execute LLM 310 on the cascading of natural language workflow queries 1702, images 402, energy spectra 404, current health status 406, relevant document sets 702, inference task result sets 704, or simulation result sets 706, and such execution can cause LLM 310 to produce natural language responses 1302 or synthetic codes 1304.

[0241] In such implementations, the natural language response 1302 may include workflow guidance 1704. In various aspects, workflow guidance 1704 may be one or more unstructured or plain text statements or sentence fragments that semantically answer the natural language workflow query 1702. That is, workflow guidance 1704 may be synthetic text that describes, states, or lists a sequence of steps (e.g., as key points) required or demanded to perform the specific microscopy workflow specified in the natural language workflow query 1702. In various cases, some microscopy workflows may be sample-dependent. Thus, workflow guidance 1704 may be considered to describe or teach what specific sequence of steps is needed or required to correctly perform the specified microscopy workflow on the currently loaded sample 306, even though the natural language workflow query 1702 may not relate to the currently loaded sample 306 at all.

[0242] As a non-limiting example, suppose natural language workflow query 1702 queries voltage comparison analysis. Furthermore, suppose voltage comparison analysis for an organic sample might require sequential execution of steps A, B, and C, and suppose voltage comparison analysis for an inorganic sample might require sequential execution of steps D, B, E, and C. As mentioned above, image 402 or energy spectrum 404 can be considered to convey at least some information about the chemical, physical, or compositional properties of the currently loaded sample 306. Because LLM 310 can be tuned based on image 402 or energy spectrum 404, LLM 310 can be considered to have access to this chemical, physical, or compositional information. Therefore, LLM 310 can infer or predict whether the currently loaded sample 306 is organic or inorganic, and LLM 310 can accordingly synthesize workflow guidance 1704 to state, describe, or list the appropriate sequence of voltage comparison analysis steps (e.g., if LLM 310 infers that the currently loaded sample 306 is organic, workflow guidance 1704 can teach the sequence of steps A, B, and C; conversely, if LLM 310 infers that the currently loaded sample 306 is inorganic, workflow guidance 1704 can alternatively teach the sequence of steps D, B, E, and C). As above, it should be noted that the related document set 702, the inference task results set 704, or the simulation results set 706 can be considered as additional, supplementary, contextual, or otherwise enriching information that assists or helps LLM 310 in correctly or accurately synthesizing workflow guidance 1704 (e.g., correctly or accurately inferring or predicting what sequence of steps is required or demanded for correctly performing the specified microscopy workflow on the currently loaded sample 306).

[0243] In some cases, it may be possible (e.g., due to a lack of training, skill, or experience of the user or technician) that the workflow specified in the Natural Language Workflow Query 1702 may be inappropriate or otherwise unexecutable for the currently loaded sample 306 (e.g., some workflows may not be applicable to samples containing certain chemicals or exhibiting certain physical structures). In such cases, the LLM 310 may infer or predict (due to any data in the image 402, energy dispersive spectroscopy 404, or their supplemental or background data) that the specified microscopy workflow cannot be correctly performed on the currently loaded sample 306, and the workflow guidance 1704 may state or describe this. In various respects, the LLM 310 may infer or predict (again, due to any data in the image 402, energy dispersive spectroscopy 404, or their supplemental or background data) that some alternative microscopy workflows can be performed on or otherwise applied to the currently loaded sample 306. In such cases, the workflow guidance 1704 may describe or explain any sequence of steps required or demanded for correctly performing the alternative microscopy workflow on the currently loaded sample 306.

[0244] In various respects, as mentioned above, the LLM 310 can generate synthetic code 1304 in addition to (or in some cases, in place of) the natural language response 1302. Also as mentioned above, synthetic code 1304 can be one or more lines of computer code capable of performing (at execution, runtime, or compilation) any suitable computerized action concerning the charged particle microscope 302 or the natural language response 1302. As a non-limiting example, where the natural language response 1302 includes workflow guidance 1704, execution of synthetic code 1304 can cause the electronic display of the charged particle microscope 302 (or any suitable computerized workstation associated with the charged particle microscope 302) to play one or more pre-recorded video graphic visualizations or animations associated with workflow guidance 1704. For example, suppose workflow guidance 1704 explains or teaches in textual terms how to correctly perform voltage contrast analysis on the currently loaded sample 306, and suppose there exists a database or repository of pre-recorded videos or animations for various microscopy workflows. In this context, LLM 310 can be considered to identify which specific videos or animations in the database or repository correspond to any specific step of voltage comparison analysis that is inferred to be appropriate or necessary for the currently loaded sample 306. Therefore, LLM 310 can cause synthesis code 1304 to reference or otherwise invoke those identified videos or animations, such that execution of synthesis code 1304 enables the visual playback of those identified videos or animations on one or more computer screens (e.g., each video or animation can visually demonstrate how one or more corresponding steps of voltage comparison analysis are performed).

[0245] As described above, the presenter component 330 can electronically perform any appropriate action regarding the natural language response 1302 or the synthetic code 1304, such as visually presenting or audibly playing the natural language response 1302. In some cases, the presenter component 330 can electronically execute, run, or compile the synthetic code 1304 to play any pre-recorded video or animation corresponding to the workflow guidance 1704. Therefore, even if the user or technician is inexperienced or untrained in the field of charged particle microscopy, they can still competently perform a microscopy workflow on or using the charged particle microscope 302 in a manner suitable for the currently loaded sample 306. In this way, the system 316 can be considered to provide the user or technician with real-time, sample-adjusted workflow guidance.

[0246] Figures 18 to 19 Example non-limiting flowcharts illustrating computer-implemented methods according to one or more embodiments described herein are provided.

[0247] First, consider Figure 18 In various embodiments, action 1802 may include access by a device (e.g., via 322) operatively coupled to a processor (e.g., 318) to a natural language request (e.g., 1702) inquiring how to perform a given microscopy workflow using a charged particle microscope (e.g., 302).

[0248] In various aspects, as shown in the figure, the computer-implemented method 1800 can proceed to actions 1504, 1506, 1508, and 1510 as described above. The computer-implemented method 1800 can then proceed to action 1602 of the computer-implemented method 1900.

[0249] Now, consider Figure 19 In various cases, action 1602 may be as described above, and the computer-implemented method 1900 may proceed to action 1902. In various cases, action 1902 may include the execution of a large language model (e.g., 310) by the device (e.g., via 328) on a natural language request (e.g., 1702), on an image (e.g., 402) or energy spectrum (e.g., 404), on a current health status (e.g., 406), on one or more documents (e.g., 702), on one or more inference task results (e.g., 704), or on one or more simulation results (e.g., 706). In various cases, this may result in the large language model producing a natural language response (e.g., including 1302 of 1704) that explains, describes, or teaches how to correctly perform a given microscopy workflow with respect to the currently loaded sample (e.g., 306).

[0250] As shown in the figure, the computer-implemented method 1900 can perform the action 1606 as described above.

[0251] Next, consider Figure 20 Unlike providing natural language commands 308, users or technicians of the charged particle microscope 302 can alternatively provide natural language troubleshooting 2002.

[0252] In various respects, a natural language fault query 2002 can be any suitable number of plain text or unstructured sentences or sentence fragments that request or command why the charged particle microscope 302 is experiencing any suitable microscopy malfunction symptom. In some cases, a natural language fault query 2002 may include a sample description 2004, which may be one or more unstructured or plain text sentences or sentence fragments that identify at least some known aspects, details, features, or properties of the currently loaded sample 306. As some non-limiting examples, a natural language fault query 2002 may be any of the following: “I have a steel sample. Why is my image blurry?”; “I have an organic sample. Why is the microscope emitting error code ABC?”; or “Tell me why the microscope is beeping on my printed circuit board sample.”

[0253] As described above, the natural language fault query 2002 can be provided or entered by a user or technician via any suitable human-machine interface device associated with the charged particle microscope 302.

[0254] As above, the status component 324 may electronically acquire, receive, retrieve, or otherwise access the image 402, energy spectrum 404, or current health status 406 in response to the receipt or access of the natural language fault query 2002.

[0255] As above, the context component 326 can electronically acquire, receive, retrieve, or otherwise access the set of inference task results 704 by executing the set of available deep learning models 1002 on the image 402 or the energy spectrum 404, respectively.

[0256] In various respects, context component 326 may be as about Figures 8 to 9 The description of obtaining, receiving, retrieving, or otherwise accessing a collection of related documents electronically 702, but in which natural language instructions 308 are replaced by natural language fault queries 2002.

[0257] Similarly, in various situations, context component 326 can be as about Figures 11 to 12 The description refers to acquiring, receiving, retrieving, or otherwise accessing a set of simulation results 706 electronically, but in which natural language instructions 308 are replaced by natural language fault queries 2002. In some such cases, digital twin prompts 1102 may be sentences such as: “The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by current health status 406. q different experiments are designed for digital twin 602 to investigate the fault symptoms specified in natural language fault queries 2002.”

[0258] Furthermore, in some aspects, a microscopy diagnostic prompt 2008 may exist. In various aspects, the microscopy diagnostic prompt 2008 may be one or more unstructured or plain text sentences or sentence fragments that request or command the execution of a self-diagnostic protocol or test (e.g., automated lens or aperture functionality test, automated electron gun functionality test) of the charged particle microscope 302. As a non-limiting example, for any suitable positive integer r, the microscopy diagnostic prompt 2008 may be the following sentence: “The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by its current health status 406. Instruct the microscope to run r most relevant self-diagnostic checks to investigate the fault specified in the natural language fault query 2002.”

[0259] Therefore, in various cases, context component 326 can electronically execute LLM 310 on natural language fault query 2002, image 402, energy spectrum 404, current health status 406, microscopy diagnostic tips 2008, or any suitable combination thereof. In various cases, such execution can cause LLM 310 to generate a set of microscopy self-diagnostic function calls 2006. More specifically, context component 326 can cascade natural language fault query 2002, image 402, energy spectrum 404, current health status 406, microscopy diagnostic tips 2008, or any combination thereof, and context component 326 can feed this cascade to the input layer of encoder section 312. In various aspects, this cascade can complete the forward pass through one or more hidden layers of encoder section 312. In various cases, the output layer of encoder section 312 can compute or otherwise compute one or more embeddings (not shown) based on activation maps or feature maps provided by one or more hidden layers of encoder section 312. In various cases, the one or more embeddings can be routed to the input layer of the synthesizer section 314. In various aspects, the one or more embeddings can perform forward passes through one or more hidden layers of the synthesizer section 314, and the output layer of the synthesizer section 314 can compute or otherwise compute the microscope self-diagnostic function call set 2006 based on the activation map or feature map provided by one or more hidden layers of the synthesizer section 314.

[0260] In various cases, as mentioned above, the microscopy diagnostic prompt 2008 may request or command the charged particle microscope 302 to perform r different self-diagnostic protocols, tests, or checks. Therefore, the microscopy self-diagnostic function call set 2006 may include r calls: microscopy self-diagnostic function call 2006(1) to microscopy self-diagnostic function call 2006(r). In various cases, each microscopy self-diagnostic function call in the microscopy self-diagnostic function call set 2006 may be considered as one or more corresponding synthetic computer code lines that can be read by and invoked or activated by the charged particle microscope 302 (e.g., in any syntax understood by the charged particle microscope 302, such as JSON code or Python code) to perform a corresponding self-diagnostic check or test. As a non-limiting example, microscopy self-diagnostic function call 2006(1) may be one or more first synthetic computer code lines that invoke a first self-diagnostic check (e.g., an automated focusing lens self-check protocol) that can be run on the charged particle microscope 302. As another non-limiting example, the microscope self-diagnostic function call 2006(r) can be one or more r-th synthetic computer code lines that define an r-th self-diagnostic check (e.g., an automated gas injector self-check protocol) that can be run on the charged particle microscope 302.

[0261] Next, consider Figure 21 In various implementations, context component 326 may electronically instruct, command, or otherwise cause charged particle microscope 302 to run, execute, or execute each microscope self-diagnostic function call in microscope self-diagnostic function call set 2006, thereby generating self-diagnostic result set 2102.

[0262] As a non-limiting example, context component 326 may enable charged particle microscope 302 to execute microscope self-diagnostic function call 2006(1). That is, context component 326 may enable charged particle microscope 302 to automatically initiate, execute, or perform a first self-diagnostic check defined by microscope self-diagnostic function call 2006(1). In various respects, such first self-diagnostic check may produce a self-diagnostic result 2102(1), wherein the self-diagnostic result 2102(1) may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, or any suitable combination thereof, measured or returned by charged particle microscope 302 during or in response to the execution of the first self-diagnostic check.

[0263] As another non-limiting example, context component 326 may enable charged particle microscope 302 to execute microscope self-diagnostic function call 2006(r). That is, context component 326 may enable charged particle microscope 302 to automatically initiate, execute, or perform the r-th self-diagnostic check defined by microscope self-diagnostic function call 2006(r). As above, such r-th self-diagnostic check may produce a self-diagnostic result 2102(r), wherein the self-diagnostic result 2102(r) may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, or any suitable combination thereof, measured or returned by charged particle microscope 302 during or in response to the execution of the r-th self-diagnostic check.

[0264] In all cases, self-diagnostic results 2102(1) to self-diagnostic results 2102(r) can be collectively considered as the set of self-diagnostic results 2102.

[0265] Now, consider Figure 22 In various respects, model component 328 can electronically execute LLM 310 on a cascade of natural language fault queries 2002, images 402, energy spectra 404, current health status 406, relevant document sets 702, inference task result sets 704, simulation result sets 706, or self-diagnostic result sets 2102, and such execution can cause LLM 310 to generate natural language responses 1302 or synthetic codes 1304.

[0266] In such implementations, the natural language response 1302 may include fault diagnosis 2202. In various aspects, fault diagnosis 2202 may be one or more unstructured or plain text statements or sentence fragments that semantically answer the natural language fault query 2002. That is, fault diagnosis 2202 may be synthetic text describing, stating, or explaining what appears to be causing the charged particle microscope 302 to experience a specific microscopy fault symptom specified in the natural language fault query 2002. In various cases, sample description 2004 may be considered known information about the currently loaded sample 306. In contrast, image 402 or energy spectrum 404 (or any information in their supplementary or background information, such as 702, 704, 706, or 2102) may be considered measured or detected information about the currently loaded sample 306. In some cases, inconsistencies, mismatches, or other interrelationships between this known information and the measured or detected information may inform or otherwise suggest what might be causing a specific problem with the charged particle microscope 302. Therefore, LLM 310 can be considered to utilize such inconsistencies, mismatches, or interrelationships to infer or predict the cause of a specific fault symptom specified in the Natural Language Fault Query 2002. In various cases, Fault Diagnosis 2202 can not only describe or explain the cause of any fault symptom specified in the Natural Language Fault Query 2002, but also describe or explain what sequence of steps should be taken to resolve such fault symptom.

[0267] As a non-limiting example, suppose natural language fault query 2002 inquires why charged particle microscope 302 is generating a blurry image of an organic sample. Furthermore, suppose the organic sample must first be dehydrated before analysis by charged particle microscope to avoid condensation artifacts. As mentioned above, the sample description 2004 in this non-limiting example may indicate that the currently loaded sample 306 is organic. Because LLM 310 can be adjusted based on this information, LLM 310 can infer or predict that the currently loaded sample 306 needs dehydration before microscopic analysis. Now, in some cases, image 402 or energy spectrum 404 may contain artifacts or anomalies that are typically, frequently, or otherwise often associated with lens condensation. Therefore, LLM 310 can determine or conclude that the currently loaded sample 306 has not received its required dehydration before analysis. Therefore, LLM 310 can accordingly synthesize fault diagnosis 2202 to state or explain that the failure to dehydrate the currently loaded sample 306 resulted in the specific microscopic fault specified in natural language fault query 2002. In some cases, fault diagnosis 2202 can further explain how such a fault can be resolved (e.g., by wiping or drying the lens of the charged particle microscope 302, and by applying a dehydration protocol to the currently loaded sample 306). As above, it should be noted that the relevant document set 702, the inference task result set 704, the simulation result set 706, or the self-diagnosis result set 2102 can be considered as additional, supplementary, contextual, or otherwise enriching information that assists or helps the LLM 310 in correctly or accurately synthesizing fault diagnosis 2202 (e.g., correctly or accurately inferring or predicting what is causing a particular fault symptom or how to resolve that particular fault symptom).

[0268] In various respects, as mentioned above, the LLM 310 can generate synthetic code 1304 in addition to (or in some cases, in place of) the natural language response 1302. Also as mentioned above, synthetic code 1304 can be one or more lines of computer code capable of performing (at execution, runtime, or compilation) any suitable computerized action concerning the charged particle microscope 302 or the natural language response 1302. As a non-limiting example, in the case where the natural language response 1302 includes fault diagnosis 2202, the LLM 310 can infer or predict that the sequential execution of one or more automated actions of the charged particle microscope 302 (e.g., automated lens drying or wiping, automated electron gun restart, automated gas injector cycling) may resolve or correct the fault symptom specified in the natural language fault query 2002. In such cases, execution of synthetic code 1304 can cause the charged particle microscope 302 to perform such a sequence of automated actions, thereby resolving the specific fault symptom. However, in other cases, LLM 310 may alternatively infer or predict the need for manual intervention or maintenance to address specific fault symptoms. In such cases, execution of Synthetic Code 1304 may schedule or otherwise mark such manual intervention or service (e.g., automatically inserting a maintenance request for the charged particle microscope 302 into an electronic maintenance calendar or database).

[0269] As described above, the presenter component 330 can electronically perform any appropriate action regarding the natural language response 1302 or the synthetic code 1304, such as visually presenting or audibly playing the natural language response 1302. In some cases, the presenter component 330 can electronically execute, run, or compile the synthetic code 1304 to automatically resolve specific fault symptoms or otherwise automatically schedule maintenance calls to resolve specific fault symptoms. Therefore, even if the user or technician is inexperienced or untrained in the field of charged particle microscopy, the user or technician can still competently resolve or handle faults in the charged particle microscope 302. In this way, the system 316 can be considered to provide the user or technician with real-time, sample-adjusted fault detection guidance.

[0270] Figures 23 to 24 Example non-limiting flowcharts illustrating computer-implemented methods according to one or more embodiments described herein are provided.

[0271] First, consider Figure 23 In various embodiments, action 2302 may include a natural language request (e.g., 2002) by a device (e.g., via 322) operatively coupled to a processor (e.g., 318) to inquire about a fault in the charged particle microscope (e.g., 302) and to describe at least in part the currently loaded sample (e.g., 306) of the charged particle microscope.

[0272] In various aspects, as shown in the figure, the computer-implemented method 2300 can then proceed to actions 1504, 1506, 1508, and 1510, as described above. The computer-implemented method 2300 can then proceed to action 1602 of the computer-implemented method 2400.

[0273] Now, consider Figure 24 In various cases, action 1602 may be performed as described above, and the computer-implemented method 2400 may proceed to action 2402. In various cases, action 2402 may include the device (e.g., via 326) causing the charged particle microscope to perform one or more self-diagnostic tests based on one or more function calls (e.g., 2006) generated by a large language model (e.g., 310) based on natural language requests, images or energy spectra, or the current health status. In various cases, this may cause the charged particle microscope to produce one or more self-diagnostic results (e.g., 2102).

[0274] In various respects, action 2404 may include the execution of a large language model by the device (e.g., via 328) on natural language requests (e.g., 2002), on images (e.g., 402) or energy spectra (e.g., 404), on current health status (e.g., 406), on one or more documents (e.g., 702), on one or more inference task results (e.g., 704), on one or more simulation results (e.g., 706), or on one or more self-diagnostic results (e.g., 2102). In various cases, this may result in the large language model producing a natural language response (e.g., including 1302 of 2202) that explains, describes, or teaches why or how the charged particle microscope is malfunctioning.

[0275] As shown in the figure, the computer-implemented method 2400 can then proceed to action 1606.

[0276] Now, consider Figure 25 Unlike providing natural language instructions 308, users or technicians of the charged particle microscope 302 can alternatively provide natural language sample queries 2502.

[0277] In various respects, a natural language sample query 2502 can be any suitable number of plain text or unstructured sentences or sentence fragments that request or command any suitable physical, chemical, or compositional property of the currently loaded sample 306. As some non-limiting examples, a natural language sample query 2502 can be any of the following: “What is the chemical composition of this sample?”; “Tell me the crystal structure of this sample”; or “What is the melting point of this sample?”.

[0278] As described above, the natural language sample query 2502 can be provided or entered by the user or technician of the charged particle microscope 302 via any suitable human-machine interface device associated with the charged particle microscope 302.

[0279] As above, the status component 324 may electronically acquire, receive, retrieve, or otherwise access the image 402, energy spectrum 404, or current health status 406 in response to the receipt or access of the natural language sample query 2502.

[0280] As above, the context component 326 can electronically acquire, receive, retrieve, or otherwise access the set of inference task results 704 by executing the set of available deep learning models 1002 on the image 402 or the energy spectrum 404, respectively.

[0281] In various respects, context component 326 may be as about Figures 8 to 9 The description of obtaining, receiving, retrieving, or otherwise accessing a relevant collection of documents electronically 702, but in which natural language instructions 308 are replaced by natural language sample queries 2502.

[0282] Similarly, in various situations, context component 326 can be as about Figures 11 to 12 The description refers to electronically acquiring, receiving, retrieving, or otherwise accessing a set of simulation results 706, but in which natural language instructions 308 are replaced by natural language sample queries 2502. In some cases of this, the digital twin prompt 1102 may be a sentence like: "The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by current health status 406. q different experiments are designed for the digital twin 602 to optimally determine the sample attributes identified in the natural language sample query 2502."

[0283] Therefore, in various respects, model component 328 can electronically execute LLM 310 on a cascade of natural language sample queries 2502, images 402, energy spectra 404, current health status 406, relevant document sets 702, inference task result sets 704, or simulation result sets 706, and such execution can cause LLM 310 to produce natural language responses 1302 or synthetic codes 1304.

[0284] In such implementations, the natural language response 1302 may include a sample interpretation 2504. In various aspects, the sample interpretation 2504 may be one or more unstructured or plain text statements or sentence fragments that semantically answer the natural language sample query 2502. That is, the sample interpretation 2504 may be synthetic text that describes, states, or lists any physical, chemical, or compositional properties of the currently loaded sample 306 requested by the natural language sample query 2502 (e.g., as a summary). In fact, as mentioned above, the image 402 or the energy spectrum 404 may be considered to convey at least some information about the chemical, physical, or compositional properties of the currently loaded sample 306. Because the LLM 310 may be modulated based on the image 402 or the energy spectrum 404, the LLM 310 may be considered to have access to this chemical, physical, or compositional information and may use this information to infer or predict the answer to the natural language sample query 2502. LLM 310 can therefore synthesize sample interpretation 2504 to state, describe, or list any properties, characteristics, or attributes of the currently loaded sample 306 queried by natural language sample query 2502. As above, it should be noted that the relevant document set 702, the inference task result set 704, or the simulation result set 706 may be considered as additional, supplementary, contextual, or otherwise enriching information that assists or helps LLM 310 in correctly or accurately synthesizing sample interpretation 2504 (e.g., correctly or accurately inferring or predicting the attributes of the requested currently loaded sample 306).

[0285] In various respects, as mentioned above, the LLM 310 may generate synthetic code 1304 in addition to (or in some cases, in lieu of) the natural language response 1302. Also as mentioned above, synthetic code 1304 may be one or more lines of computer code capable of performing (at execution, runtime, or compilation) any suitable computerized action concerning the charged particle microscope 302 or the natural language response 1302. As a non-limiting example, in the case where the natural language response 1302 includes sample interpretation 2504, execution of synthetic code 1304 may cause the electronic display of the charged particle microscope 302 (or any suitable computerized workstation associated with the charged particle microscope 302) to display or present any suitable plots, tables, or other graphs showing any properties, characteristics, or attributes of the currently loaded sample 306 that the LLM 310 has measured or inferred.

[0286] As described above, the presenter component 330 can electronically perform any appropriate action regarding the natural language response 1302 or the synthetic code 1304, such as visually presenting or audibly playing the natural language response 1302. In some cases, the presenter component 330 can electronically execute, run, or compile the synthetic code 1304 to display various charts, tables, graphs, or plots about the currently loaded sample 306. Therefore, even if the user or technician is inexperienced or untrained in the field of charged particle microscopy, the user or technician can still competently analyze the currently loaded sample 306 using the charged particle microscope 302. In this way, the system 316 can be considered to provide users or technicians with real-time, automated sample analysis.

[0287] Figures 26 to 27 Example non-limiting flowcharts illustrating computer-implemented methods according to one or more embodiments described herein are provided.

[0288] First, consider Figure 26 In various embodiments, action 2602 may include access by a device (e.g., via 322) operatively coupled to a processor (e.g., 318) to a natural language request (e.g., 2502) to inquire about one or more unknown properties of a currently loaded sample (e.g., 306) of a charged particle microscope (e.g., 302).

[0289] In various aspects, as shown in the figure, the computer-implemented method 2600 can proceed to actions 1504, 1506, 1508, and 1510 as described above. The computer-implemented method 2600 can then proceed to action 1602 of the computer-implemented method 2700.

[0290] Now, consider Figure 27 In various cases, action 1602 may be performed as described above, and the computer-implemented method 2700 may proceed to action 2702. In various cases, action 2702 may include the execution of a large language model (e.g., 310) by the device (e.g., via 328) on a natural language request (e.g., 1702), on an image (e.g., 402) or energy spectrum (e.g., 404), on a current health status (e.g., 406), on one or more documents (e.g., 702), on one or more inference task results (e.g., 704), or on one or more simulation results (e.g., 706). In various cases, this may result in the large language model producing a natural language response (e.g., including 1302 of 2504) that identifies, interprets, or describes one or more unknown characteristics of the currently loaded sample.

[0291] As shown in the figure, the computer-implemented method 2600 can then proceed to action 1606.

[0292] Next, consider Figure 28 In various respects, the user or technician of the charged particle microscope 302 may have previously provided (e.g., typed or spoken) multiple past natural language queries 2802 to the charged particle microscope 302. In various cases, the multiple past natural language queries 2802 may include any suitable number of past natural language queries. In various cases, past natural language queries may be any suitable unstructured or plain text query as described herein (e.g., an instruction to set any of 304 to a specific value; a workflow query; a fault query; a sample query).

[0293] As described above, the status component 324 may electronically acquire, receive, retrieve, or otherwise access the image 402, energy spectrum 404, or current health status 406.

[0294] As above, the context component 326 can electronically acquire, receive, retrieve, or otherwise access the set of inference task results 704 by executing the set of available deep learning models 1002 on the image 402 or the energy spectrum 404, respectively.

[0295] As above, context component 326 can perform an appropriate embedding search via document repository 902 (e.g., through...). Figure 8 The Chinese uses 2802 instead of 308 to obtain, receive, retrieve or otherwise access a collection of related documents electronically.

[0296] As above, context component 326 can generate appropriate function calls to digital twin 602 using LLM 310 (e.g., through...). Figure 11 (Use 2802 instead of 308) to obtain, receive, retrieve or otherwise access the set of simulation results 706 electronically.

[0297] Currently, a GUI prompt 2804 may exist in various aspects. In various cases, the GUI prompt 2804 may be one or more unstructured or plain text sentences or sentence fragments that request or command to construct a graphical user interface (GUI) for both the charged particle microscope 302 and the currently loaded sample 306, suitable for both the user or technician. As a non-limiting example, the GUI prompt 2804 may be the following sentence: "The sample is characterized by image 402 or energy spectrum 404. The microscope is characterized by its current health status 406. Create a microscope GUI that is relevant to the sample and suitable for a user who has already queried multiple past natural language queries 2802."

[0298] Therefore, in various respects, model component 328 can electronically execute LLM 310 on a cascade of multiple past natural language queries 2802, images 402, energy spectra 404, current health status 406, relevant document sets 702, inference task result sets 704, or simulation result sets 706, and such execution can cause LLM 310 to generate synthetic code 1304 (in some embodiments, natural language response 1302 may be omitted).

[0299] In various respects, the synthetic code 1304 may be one or more lines of computer code capable of performing any suitable computerized action concerning the charged particle microscope 302 (at execution, runtime, or compilation time). As a non-limiting example, the synthetic code 1304 may define or otherwise serve as source code or programming scripts for sample customization and user-customized GUIs for the charged particle microscope 302.

[0300] Specifically, it is possible that each of the multiple configurable operating settings 304 is relevant or suitable for some types of samples, but irrelevant or unsuitable for others. For example, a domain of a possible microscopy protocol or setting that can be nondestructively or effectively applied to inorganic samples may differ from or otherwise differ from a domain of a possible microscopy protocol or setting that can be nondestructively or effectively applied to organic samples. Similarly, a domain of a possible microscopy protocol or setting that can be nondestructively or effectively applied to metal samples may differ from or otherwise differ from a domain of a possible microscopy protocol or setting that can be nondestructively or effectively applied to plastic samples. In any case, image 402 or energy dispersive spectroscopy 404 may be considered to convey at least some physical, chemical, or compositional information about the currently loaded sample 306. Because the LLM 310 can be adjusted based on image 402 or energy spectrum 404, it can use this information to infer or predict which of the multiple configurable operating settings 304 are relevant or suitable for the currently loaded sample 306, and which are not. Therefore, the LLM 310 can create synthesis code 1304 that defines or creates a GUI that includes or displays only those configurable operating settings 304 that are relevant or suitable for the currently loaded sample 306, and excludes or does not display the operating settings 304 that are irrelevant or unsuitable for the currently loaded sample 306. In this way, the GUI defined by synthesis code 1304 can be considered customized or tailored for the currently loaded sample 306.

[0301] Additionally, it is possible that each of the multiple configurable operating settings 304 may be too advanced or complex for inexperienced, untrained, or unskilled users or technicians. In various cases, multiple past natural language queries 2802 may be considered to implicitly convey or capture the user's or technician's level of microscopy skill. For example, a history of querying more basic or simple queries may indicate that the user or technician has less microscopy skill, while a history of querying more complex or cumbersome queries may indicate that the user or technician has, alternatively, more microscopy skill. Because LLM 310 can be tuned based on multiple past natural language queries 2802, LLM 310 can use this implicit information to infer or predict the user's or technician's level of microscopy skill or experience. Therefore, LLM 310 can identify which of the multiple configurable operating settings 304 are proportionate or appropriate to the inferred level of microscopy skill or experience, and which configurable operating settings are too advanced for the inferred level of microscopy skill or experience. Therefore, the LLM310 can create synthesis code 1304 such that it defines or creates a GUI that includes or displays only those configurable operating settings 304 that are appropriate or not too complex for the inferred level of microscopy technique or experience of the user or technician, and does not include or display those configurable operating settings 304 that are inappropriate or too complex for the inferred level of microscopy technique or experience of the user or technician. In this way, the GUI defined by synthesis code 1304 can be considered customized or tailored for the user or technician.

[0302] Furthermore, given a current health state 406, it is possible that individual configurable operating settings among multiple configurable operating settings 304 are currently unsafe to invoke, activate, or change. For example, some components of the charged particle microscope 302 may be too worn to properly execute certain protocols or safely adjust certain settings. Because the LLM 310 can be adjusted based on the current health state 406, the LLM 310 can infer or predict which configurable operating settings among the multiple configurable operating settings 304 are currently unsafe to invoke, activate, or change. Therefore, the LLM 310 can create synthetic code 1304 that defines or creates a GUI that includes or displays only those configurable operating settings among the multiple configurable operating settings 304 that are currently safe to invoke, activate, or change, and does not include or display the operating settings among the multiple configurable operating settings 304 that are currently unsafe to invoke, activate, or change. In this way, the GUI defined by synthetic code 1304 can be considered customized or tailored based on the current health state of the charged particle microscope 302.

[0303] As described above, the presenter component 330 can electronically perform any appropriate action regarding the synthesis code 1304. In some cases, the presenter component 330 can electronically execute, run, or compile the synthesis code 1304 to present or activate a customized or tailored GUI for the charged particle microscope 302. Therefore, a wide variety of users or technicians can interact with the charged particle microscope 302 intuitively or comfortably, regardless of their specific or different levels of microscopy expertise.

[0304] Figures 29 to 30 Example non-limiting flowcharts illustrating computer-implemented methods according to one or more embodiments described herein are provided.

[0305] First, consider Figure 29 In various implementations, action 2902 may include access by a device (e.g., via 322) operatively coupled to a processor (e.g., 318) to a plurality of past natural language requests (e.g., 2802) queried by a user of a charged particle microscope (e.g., 302).

[0306] In various respects, action 2904 may include the device (e.g., via 324) causing the charged particle microscope to capture an image (e.g., 402) or energy spectrum (e.g., 404) of the currently loaded sample (e.g., 306) using a default microscopy protocol.

[0307] In various cases, action 2906 may include accessing the current health status of the charged particle microscope by the device (e.g., via 324) and from a digital twin synchronized with the charged particle microscope (e.g., 602) (e.g., 406).

[0308] In various cases, action 2908 may include identifying one or more documents (e.g., 702) related to multiple past natural language requests, images or energy spectra, or current health status by the device (e.g., via 326) and via an embedded search of a document database (e.g., 902).

[0309] In various respects, the computer-implemented method 2900 can then proceed to the action 1510 as described above, and then proceed to the action 3002 of the computer-implemented method 3002.

[0310] Now consider Figure 30In various embodiments, action 3002 may include the device (e.g., via 326) operating the digital twin based on one or more function calls (e.g., 1104) generated by a large language model (e.g., 310) based on multiple past natural language requests, based on images or energy spectra, or based on the current health status. In various cases, this may cause the digital twin to produce one or more simulation results (e.g., 706).

[0311] In various respects, action 3004 may include the device (e.g., via 328) performing a large language model on multiple past natural language requests, on images or energy spectra, on current health status, on one or more documents, on one or more inference task results, or on one or more simulation results. In various cases, this may result in the large language model generating synthetic code (e.g., 1304) that defines a graphical user interface for the charged particle microscope tailored to the currently loaded sample and to the inferred level of user experience.

[0312] In various cases, action 3006 may include synthesized code executed by the device (e.g., via 330). In various cases, this may activate a graphical user interface or otherwise present a graphical user interface to a user.

[0313] To ensure that the natural language response 1302 or the synthesized code 1304 is accurate or correct according to the various implementation schemes described herein, the LLM 310 (and the available set of deep learning models 1002) may first undergo training. Regarding Figure 31 An unrestricted example of this training is described.

[0314] Figure 31 Example non-limiting block diagrams illustrating how an LLM 310 (or each of the available deep learning models in the set of 1002 available deep learning models) can be trained according to one or more embodiments described herein are shown.

[0315] In various respects, before training begins, the trainable intrinsic parameters (e.g., convolutional kernels, weight matrices, biases) of the LLM 310 (or any deep learning model in the deep learning model set 1002) can be initialized by the system 316 in any suitable manner (e.g., via random initialization).

[0316] In various implementations, training input 3102 and baseline truth annotation 3104 may exist. When it is desired to train the LLM310, training input 3102 can be any suitable text, numerical, or graphical data that can be received by the LLM310, as described herein. As some examples, which are not limiting by any means, the training input 3102 can be any suitable natural language instruction, question, command, or query (e.g., any of 308, 1702, 2002, 2502, or 2802), which may or may not be cascaded with any suitable image (e.g., such as 402), any suitable energy spectrum (e.g., such as 404), any suitable microscope health status (e.g., such as 406), any suitable relevant document (e.g., such as 702), any suitable inference task result (e.g., such as 704), any suitable digital twin simulation result (e.g., such as 706), any suitable microscope self-diagnosis result (e.g., such as 2102), or any suitable supplementary prompt (e.g., such as 1102, 1401, 2008, or 2804). In this case, the benchmark truth annotation 3104 can be any correct or accurate synthetic text content (e.g., such as 1302) or code (e.g., such as 1304) that is known or considered to correspond to the training input 3102. Conversely, when it is desired to train any available deep learning model in the set of available deep learning models 1002, the training input 3102 can be any suitable training image or training energy spectrum (or their cascade), and the benchmark ground truth annotation 3104 can be any correct or accurate inference task result known or considered to correspond to the training input 3102 (e.g., correct or accurate classification label, correct or accurate segmentation mask, correct or accurate regression output).

[0317] In any case, system 316 may enable the execution of LLM 310 (or any available deep learning model in the set of available deep learning models 1002) on training input 3102, thereby causing LLM 310 (or any available deep learning model in the set of available deep learning models 1002) to produce output 3106. More specifically, in some cases, training input 3102 may be fed or routed to the input layer of LLM 310 (or any available deep learning model in the set of available deep learning models 1002), training input 3102 may perform forward passes through one or more hidden layers of LLM 310 (or any available deep learning model in the set of available deep learning models 1002), and the output layer of LLM 310 (or any available deep learning model in the set of available deep learning models 1002) may compute output 3106 based on activation maps or feature maps provided by one or more hidden layers of LLM 310 (or any available deep learning model in the set of available deep learning models 1002).

[0318] It should be noted that the format, size, or dimension of output 3106 can be specified by the number, arrangement, size, or other characteristics of neurons, convolutional kernels, attention blocks, or other internal parameters of the output layer (or any other layer) of LLM 310 (or any available deep learning model in the set of 1002 available deep learning models). Therefore, output 3106 can be forced to have any desired format, size, or dimension by adding, removing, or otherwise adjusting the characteristics of the output layer (or any other layer) of LLM 310 (or any available deep learning model in the set of 1002 available deep learning models).

[0319] In various respects, if output 3106 is produced by LLM 310, then output 3106 can be considered as predicted or inferred text content synthesized by LLM 310 based on training input 3102 (e.g., predicted or inferred natural language responses, predicted or inferred synthesized code, predicted or inferred function calls). Conversely, if output 3106 is produced by any available deep learning model in the set of available deep learning models 1002, then output 3106 can be considered as predicted or inferred inference task results produced by such available deep learning model for training input 3102 (e.g., predicted or inferred classification labels, predicted or inferred segmentation masks, predicted or inferred regression outputs). In any case, the baseline truth annotation 3104 can be considered as any correct or accurate result known or considered to correspond to training input 3102. It should be noted that if LLM 310 (or any available deep learning model in the set of available deep learning models 1002) has not undergone training or has undergone very little training to date, then output 3106 may be highly inaccurate. In other words, the output 3106 can be very different from the baseline truth comment 3104.

[0320] In various aspects, the error 3108 between the output 3106 and the baseline truth annotation 3104 can be calculated by system 316 (e.g., MAE, MSE, cross-entropy error). In various cases, the trainable intrinsic parameters of LLM 310 (or any available deep learning model in the available deep learning model set 1002) can be incrementally updated based on the error 3108 via backpropagation (e.g., stochastic gradient descent).

[0321] In various cases, this execution and update process can be repeated for any appropriate number of input annotation pairs. This can ultimately lead to the trainable intrinsic parameters of the LLM 310 (or the available deep learning model set 1002) being iteratively optimized for accurately performing text or code synthesis (or for accurately performing any other suitable inference task, such as image or spectrum classification, segmentation, or regression). In various respects, any suitable training batch size, any suitable error / loss function, or any suitable training termination criterion can be utilized during this training.

[0322] Although the disclosure herein primarily describes the LLM 310 (or the available deep learning model set 1002) as trained in a supervised manner, this is merely a non-limiting example for ease of explanation and illustration. In various implementations, any other suitable training paradigm can be used to train the LLM 310 or the available deep learning model set 1002, such as unsupervised training or reinforcement learning, either jointly or non-jointly.

[0323] Although the disclosure herein primarily describes the LLM 310 as being trained or configured to synthesize the natural language response 1302 or the code 1304, these are merely non-limiting examples for ease of explanation and illustration. In various embodiments, the LLM 310 may be configured to synthesize any suitable visual graphics that complement the natural language response 1302. Non-limiting examples of such visual graphics may include: formatted reports or presentation slides based on, derived from, or illustrating the natural language response 1302; or plots, graphs, or charts based on, derived from, or illustrating the natural language response 1302. Therefore, the presenter component 330 may visually present such visual graphics in addition to or in combination with the natural language response 1302.

[0324] Although the disclosure herein primarily describes the presenter component 330 as visually or audibly presenting or displaying the natural language response 1302 (or any associated visual graphics) to a user or technician of the charged particle microscope 302, these are merely non-limiting examples for ease of explanation and illustration. In various embodiments, the presenter component 330 may electronically transmit the natural language response 1302 (or any associated visual graphics, or synthetic code 1304) to any suitable computing device associated with the charged particle microscope. As a non-limiting example, the presenter component 330 may share the natural language response 1302 (or any associated visual graphics or synthetic code 1304) with any suitable downstream software tool or application operating or in conjunction with the charged particle microscope 302 (e.g., some embodiments may involve sending the natural language response 1302 to such downstream software tool or application, rather than presenting the natural language response 1302 to a user or technician).

[0325] Although the disclosure herein primarily describes the LLM 310 as receiving natural language input (e.g., 308, 1702, 2002, 2502, 2802) provided by a user or technician of the charged particle microscope 302, these are merely non-limiting examples for ease of explanation and illustration. In various embodiments, the LLM 310 may receive natural language input synthesized by any suitable upstream generative model (e.g., in some cases, any of 308, 1702, 2002, 2502, or 2802 may be synthesized by an upstream generative artificial intelligence model rather than being manually provided by a user or technician).

[0326] It should be understood that multiple natural language inputs (e.g., multiple instances of 308, 1702, 2002, 2502, or 2802) may be provided to system 316 (e.g., by a user or operator of the charged particle microscope 302, or by any upstream generative artificial intelligence model). In some aspects, system 316 may utilize an LLM 310 as described herein to synthesize a corresponding response to each of such multiple natural language inputs in the order in which they are received. However, in other aspects, system 316 may alternatively utilize an LLM 310 as described herein to synthesize a corresponding response to each of such multiple natural language inputs in any other suitable order (e.g., in an order different from the order in which the multiple natural language inputs are received). In some cases, such multiple natural language inputs may be provided to system 316 simultaneously or otherwise substantially simultaneously with each other, such that they can be considered to collectively form a single, holistic prompt. In such cases, it should be understood that the multiple natural language inputs can be individually identified by applying any appropriate cue profiling or cue segmentation technique to such a single, holistic cue.

[0327] Although the disclosure herein primarily describes system 316 as utilizing LLM 310 in response to natural language input (e.g., 308, 1702, 2002, 2502, 2802) associated with charged particle microscope 302, these are merely non-limiting examples for ease of explanation and illustration. In various embodiments, system 316 may utilize LLM 310 to periodically or regularly monitor the state or context of charged particle microscope 302, even in the absence of any natural language input (e.g., even in the absence of 308, 1702, 2002, 2502, and 2802). As a non-limiting example, in some embodiments, state component 324 may electronically acquire image 402, energy spectrum 404, or current health status 406 in response to (e.g., via a weight sensor or pressure sensor) determining that the actuable stage of charged particle microscope 302 is not empty. In this context, context component 326 can electronically acquire, receive, retrieve, or otherwise access the inference task result set 704 by executing the available deep learning model set 1002 on image 402 or energy spectrum 404, respectively. Similarly, in this context, context component 326 can electronically acquire, receive, retrieve, or otherwise access the relevant document set 702, such as regarding... Figures 8 to 9 The description omits the natural language instruction 308. Even in this case, the context component 326 can electronically acquire, receive, retrieve, or otherwise access the simulation results set 706, as per [the previous sentence]. Figures 11 to 12 The description is provided, but the natural language instruction 308 is omitted. Therefore, model component 328 can electronically execute LLM 310 on a cascade of images 402, energy spectrum 404, current health status 406, relevant document set 702, inference task result set 704, or simulation result set 706, and such execution can cause LLM 310 to produce a natural language response 1302 or synthetic code 1304.

[0328] In such implementations, the natural language response 1302 may include one or more unstructured or plain text statements or sentence fragments that are semantically related to the image 402, energy spectrum 404, current health status 406, relevant document set 702, inference task result set 704, or simulation result set 706 in any suitable manner. In some aspects, the natural language response 1302 may be synthetic text that describes or explains any suitable steps, actions, or tasks recommended (as inferred by the LLM 310) to be performed (e.g., by a user or technician, or by the charged particle microscope 302 itself) given the state (e.g., 402, 404, or 406) or context (e.g., 702, 704, or 706) of the charged particle microscope 302.

[0329] As a non-limiting example, LLM 310 may infer that a specific microscopy workflow or protocol should be performed on the currently loaded sample 306 (e.g., to avoid sample degradation or imaging artifacts). Therefore, the natural language response 1302 may explain or describe how to perform such a specific microscopy workflow or protocol, even without any inquiry or instruction from the user or technician of the charged particle microscope 302. In some cases, synthetic code 1304 may be configured to automatically perform all or part of that specific microscopy workflow or protocol.

[0330] As another non-limiting example, LLM 310 may infer that a specific microscopy workflow or protocol should be performed on the currently loaded sample 306 (e.g., to avoid sample degradation or imaging artifacts), but LLM 310 may also infer that a necessary component of the charged particle microscope 302 (e.g., the X-ray tube) has too much wear to properly or safely perform such a specific microscopy workflow or protocol. Therefore, the natural language response 1302 can interpret the foregoing inference and may recommend maintenance of the necessary component, even without any inquiry or instruction from the user or technician of the charged particle microscope 302. In some cases, synthetic code 1304 may be configured to automatically schedule such maintenance.

[0331] The various implementations described herein can be considered as utilizing a large language model to improve the accessibility or operability of charged particle microscopy. Specifically, the various implementations may involve adjusting the LLM based on images or energy spectra captured by the charged particle microscope. This adjustment allows the LLM to perform the synthesis of text content or code for the charged particle microscope in a more informative or accurate manner than other possible methods. In other words, if the LLM receives textual prompts as input that are not accompanied by images or energy spectra captured by the charged particle microscope, the text or code synthesized by the LLM will be less accurate or less satisfactory to users or technicians of the charged particle microscope. Therefore, the image or energy spectrum adjustment of the LLM as described herein can be considered a concrete and tangible improvement in the field of charged particle microscopy operation.

[0332] The scientific instrument systems, methods, or techniques disclosed herein may include (e.g., via references herein) Figure 34 The user-local computing device 3420 under discussion interacts with human users. These interactions may include providing information to the user (e.g., about scientific instruments such as...) Figure 34 Information on the operation of the scientific instrument 3410, information about the sample being analyzed or other tests or measurements performed by the scientific instrument, information retrieved from local or remote databases or other information, or information provided to the user for inputting commands (e.g., for controlling the scientific instrument, such as...). Figure 34 The scientific instrument 3410 provides options, queries (e.g., queries to local or remote databases), or other information for controlling the analysis of data generated by the scientific instrument. In some embodiments, these interactions can be performed via a graphical user interface (GUI), which includes a display device (e.g., referenced herein). Figure 33 A visual display on a display device (3310) discussed herein, which provides output and / or prompts to the user (e.g., via reference herein). Figure 33 Input may be provided by one or more input devices included in the other I / O devices discussed 3312, such as a keyboard, mouse, trackpad, or touchscreen. The scientific instrument systems, methods, or techniques disclosed herein may include any GUI suitable for user interaction.

[0333] Figure 32 Example graphical user interfaces 3200 (hereinafter referred to as "GUI 3200") are depicted according to various embodiments and can be used to perform some or all of the supporting methods or techniques disclosed herein. In various respects, GUI 3200 may be set up in scientific instrument support systems (e.g., as referenced herein). Figure 34 The computing device of the scientific instrument support system 3400 discussed herein (e.g., referenced herein) Figure 33Any suitable electronic display for the computing device 3300 discussed herein (e.g., referenced herein). Figure 33 On the display device 3310 discussed herein, and the user or technician may use any suitable input device (e.g., the one referred to herein). Figure 33 The GUI 3200 interacts with any of the other I / O devices 3312 discussed and input technologies (e.g., cursor movement, motion capture, facial recognition, gesture detection, speech recognition, button activation).

[0334] The GUI 3200 may include a data display area 3202, a data analysis area 3204, a scientific instrument control area 3206, and a settings area 3208. Figure 32 The specific number and arrangement of the depicted areas are merely illustrative, and any number and arrangement of areas (including any desired features) can be included in other implementations of GUI 3200.

[0335] Data display area 3202 can display data generated by scientific instruments (e.g., as referenced in this article). Figure 34 The data generated by the scientific instrument 3410 under discussion.

[0336] Data analysis area 3204 can display any suitable data analysis results (e.g., the analysis data display results of data illustrated in area 3202 or other data). In some embodiments, data display area 3202 and data analysis area 3204 can be combined in GUI 3200 (e.g., including both data output from scientific instruments and some analysis of the data in a public graph or area).

[0337] Scientific instrument control area 3206 may include allowing users or technicians to control scientific instruments (e.g., as referenced herein). Figure 34 Options for the scientific instrument 3410 under discussion. For example, the scientific instrument control area 3206 may include configurable parameters for controlling the operation of such scientific instrument (e.g., configurable parameters for controlling the voltage or current of the scientific instrument, configurable parameters for controlling the internal temperature of the scientific instrument, or configurable parameters for controlling the fluid flow rate of the scientific instrument).

[0338] The settings area 3208 may include any features or functions of the GUI 3200 (or other GUI) that allow a user or technician to control or perform common computational operations on the data display area 3202 and the data analysis area 3204 (e.g., storing data on a storage device such as those referenced herein). Figure 33 On the storage device 3304 under discussion, options include sending data to another user and tagging data.

[0339] As described above, the scientific instrument module 102 can be implemented by one or more computing devices. Figure 33 This is a block diagram of a computing device 3300 that can implement some or all of the scientific instrument support methods or techniques disclosed herein, according to various embodiments. In some embodiments, the scientific instrument module 102 may be implemented by a single instance or multiple instances of the computing device 3300. Furthermore, as discussed below, the computing device 3300 (or multiple instances of the computing device) implementing the scientific instrument module 102 may be... Figure 34 A part of one or more of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440.

[0340] Computing device 3300 is illustrated as having multiple components, but any one or more of these components may be omitted or duplicated depending on the application and setup. In some embodiments, some or all of the components included in computing device 3300 may be attached to one or more motherboards and encapsulated in a housing (e.g., comprising plastic, metal, or other materials). In some embodiments, some of these components may be fabricated on a single system-on-a-chip (SoC) (e.g., the SoC may include one or more instances of processing device 3302 and one or more instances of storage device 3304). Additionally, in various embodiments, computing device 3300 may be omitted. Figure 33 One or more of the illustrated components may be included, but may also include interface circuitry (not shown) for coupling to one or more omitted components using any suitable interface (e.g., a Universal Serial Bus (USB) interface, a High Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface). For example, computing device 3300 may omit display device 3310, but may include display device interface circuitry (e.g., connector and driver circuitry) to which display device 3310 may be coupled.

[0341] Computing device 3300 may include processing device 3302 (e.g., one or more processing devices). As used herein, the term "processing device" can refer to any device or part of a device that processes electronic data from a register or memory to convert that electronic data into other electronic data that can be stored in the register or memory. Processing device 3302 may include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), cryptographic processors (dedicated processors that execute cryptographic algorithms within hardware), server processors, or any other suitable processing device.

[0342] Computing device 3300 may include storage device 3304 (e.g., one or more storage devices). Storage device 3304 may include one or more memory devices, such as random access memory (RAM) (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive bridged RAM (CBRAM) devices), hard disk drive-based memory devices, solid-state memory devices, network drives, cloud drives, or any combination of memory devices. In some embodiments, storage device 3304 may include memory sharing a die with processing device 3302. In such embodiments, the memory may be used as cache memory and may include, for example, embedded dynamic random access memory (eDRAM) or spin-transfer torque magnetic random access memory (STT-MRAM). In some embodiments, storage device 3304 may include a non-transitory computer-readable medium having instructions thereon that, when executed by one or more processing devices (e.g., processing device 3302), cause computing device 3300 to perform any suitable method or portion thereof of the methods disclosed herein.

[0343] Computing device 3300 may include interface device 3306 (e.g., one or more instances of interface device 3306). Interface device 3306 may include one or more communication chips, connectors, or other hardware and software to manage communication between computing device 3300 and other computing devices. For example, interface device 3306 may include circuitry for managing wireless communication used to transmit data to and from computing device 3300. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, techniques, and communication channels that can transmit data over a non-solid medium using modulated electromagnetic radiation. This term does not imply that the associated device does not contain any wires, although in some embodiments it may not contain any wires. The circuitry included in interface device 3306 for managing wireless communications can implement any of a variety of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.11 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendment), Long Term Evolution (LTE) projects, and any amendments, updates, and / or revisions (e.g., Advanced LTE project, Ultra Mobile Broadband (UMB) project (also known as “3GPP2”)). In some embodiments, the circuitry included in interface device 3306 for managing wireless communications can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed ​​Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. In some embodiments, the circuitry included in interface device 3306 for managing wireless communications may operate according to Enhanced Data GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, the circuitry included in interface device 3306 for managing wireless communications may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolved Data Optimization (EV-DO) and its derivatives, as well as any other wireless protocol designated as 3G, 4G, 5G, or higher. In some embodiments, interface device 3306 may include one or more antennas (e.g., one or more antenna arrays) for receiving and / or transmitting wireless communications.

[0344] In some embodiments, interface device 3306 may include circuitry for managing wired communications, such as electrical communication protocols, optical communication protocols, or any other suitable communication protocol. For example, interface device 3306 may include circuitry supporting communications based on Ethernet technology. In some embodiments, interface device 3306 may support both wireless and wired communications, or may support multiple wired communication protocols or multiple wireless communication protocols. For example, a first set of circuitry for interface device 3306 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, while a second set of circuitry for interface device 3306 may be dedicated to longer-range wireless communications such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, a first set of circuitry for interface device 3306 may be dedicated to wireless communications, while a second set of circuitry for interface device 3306 may be dedicated to wired communications.

[0345] The computing device 3300 may include a battery / power circuit 3308. The battery / power circuit 3308 may include one or more energy storage devices (e.g., batteries or capacitors) or circuitry for coupling components of the computing device 3300 to an energy source (e.g., an AC line power supply) separate from the computing device 3300.

[0346] The computing device 3300 may include a display device 3310 (e.g., multiple display devices). The display device 3310 may include any visual indicator, such as a head-up display, a computer monitor, a projector, a touch screen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display.

[0347] The computing device 3300 may include other input / output (I / O) devices 3312. For example, other I / O devices 3312 may include one or more audio output devices (e.g., speakers, headphones, earphones, alarms), one or more audio input devices (e.g., microphones or microphone arrays), positioning devices (e.g., GPS devices that communicate with a satellite-based system to receive the orientation of the computing device 3300), audio codecs, video codecs, printers, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, accelerometers, gyroscopes), image capture devices such as cameras, keyboards, cursor control devices such as mice, styluses, trackballs or touchpads, barcode readers, quick-response (QR) code readers, or radio frequency identification (RFID) readers.

[0348] The computing device 3300 may have any suitable form factor for its application and setup, such as a handheld or mobile computing device (e.g., a mobile phone, smartphone, mobile internet device, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), ultra-mobile personal computer), desktop computing device, or server computing device or other networked computing component.

[0349] One or more computing devices that implement any of the scientific instrument modules, methods or techniques disclosed herein may be part of a scientific instrument support system. Figure 34 This is a block diagram of an example scientific instrument support system 3400, according to various implementation schemes, in which some or all of the scientific instrument support methods disclosed herein can be performed. The scientific instrument modules, methods, or techniques disclosed herein (e.g., scientific instrument module 102, computer-implemented method 200, system 316, computer-implemented methods 1500-1600, 1800-1900, 2300-2400, 2600-2700, or 2900-3000) can be implemented by one or more of the scientific instrument 3410, user local computing device 3420, service local computing device 3430, or remote computing device 3440 of the scientific instrument support system 3400.

[0350] Any of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440 may include any implementation of the implementation scheme of the computing device 3300, and any of the scientific instrument 3410, the user local computing device 3420, the service local computing device 3430, or the remote computing device 3440 may take the form of any suitable implementation scheme of the implementation scheme of the computing device 3300.

[0351] Scientific instrument 3410, user local computing device 3420, service local computing device 3430, or remote computing device 3440 may each include processing device 3402, storage device 3404, and interface device 3406. Processing device 3402 may take any suitable form, including any form of processing device 3302, and the processing devices 3402 included in different devices of scientific instrument 3410, user local computing device 3420, service local computing device 3430, or remote computing device 3440 may take the same or different forms. Storage device 3404 may take any suitable form, including any form of storage device 3304, and the storage devices 3404 included in different devices of scientific instrument 3410, user local computing device 3420, service local computing device 3430, or remote computing device 3440 may take the same or different forms. Interface device 3406 may take any suitable form, including any form of interface device 3306, and the interface device 3406 included in different devices of scientific instrument 3410, user local computing device 3420, service local computing device 3430 or remote computing device 3440 may take the same or different forms.

[0352] Scientific instrument 3410, user local computing device 3420, service local computing device 3430, and remote computing device 3440 can communicate with other components of scientific instrument support system 3400 via communication path 3408. Communication path 3408 can communicatively couple interface device 3406 of different components in scientific instrument support system 3400, as shown, and can be a wired or wireless communication path (e.g., any of the communication technologies discussed herein according to reference interface device 3306). Figure 34 The depicted specific scientific instrument support system 3400 includes communication paths between each pair of devices among scientific instrument 3410, user local computing device 3420, service local computing device 3430, and remote computing device 3440. However, this specific implementation of "full connectivity" is merely exemplary, and various communication paths in communication path 3408 may not exist in various embodiments. For example, in some embodiments, service local computing device 3430 may lack a direct communication path 3408 between its interface device 3406 and the interface device 3406 of scientific instrument 3410, but may instead communicate with scientific instrument 3410 via communication path 3408 between service local computing device 3430 and user local computing device 3420 and communication path 3408 between user local computing device 3420 and scientific instrument 3410.

[0353] Scientific instrument 3410 may include any suitable scientific instrument, such as charged particle microscope 302.

[0354] User-local computing device 3420 may be a user-local computing device of scientific instrument 3410 (e.g., any embodiment of the computing device 3300). In some embodiments, user-local computing device 3420 may also be located locally to scientific instrument 3410, but this is not mandatory; for example, user-local computing device 3420 in a user's home or office may be located away from scientific instrument 3410 but communicate with it, allowing the user to use user-local computing device 3420 to control or access data from scientific instrument 3410. In some embodiments, user-local computing device 3420 may be a laptop, smartphone, or tablet device. In some embodiments, user-local computing device 3420 may be a portable computing device.

[0355] The servicing local computing device 3430 may be a computing device physically local to the servicing scientific instrument 3410 (e.g., any of the embodiments of computing device 3300). For example, the servicing local computing device 3430 may be a local device of the manufacturer of scientific instrument 3410 or a third-party service company. In some embodiments, the servicing local computing device 3430 may (e.g., via a direct communication path 3408 or via multiple “indirect” communication paths 3408, as discussed above) communicate with scientific instrument 3410, user local computing device 3420, or remote computing device 3440 to receive data regarding the operation of scientific instrument 3410, user local computing device 3420, or remote computing device 3440 (e.g., self-test results of scientific instrument 3410, calibration coefficients used by scientific instrument 3410, and measurements from sensors associated with scientific instrument 3410). In some implementations, the service local computing device 3430 may (e.g., via a direct communication path 3408 or via multiple “indirect” communication paths 3408, as discussed above) communicate with the scientific instrument 3410, the user local computing device 3420, or the remote computing device 3440 to transfer data to the scientific instrument 3410, the user local computing device 3420, or the remote computing device 3440 (e.g., updating programming instructions (such as firmware) in the scientific instrument 3410, initiating the execution of test or calibration sequences in the scientific instrument 3410, updating programming instructions (such as software) in the user local computing device 3420 or the remote computing device 3440). Users of the scientific instrument 3410 may use the scientific instrument 3410 or the user local computing device 3420 to communicate with the service local computing device 3430 to report problems with the scientific instrument 3410 or the user local computing device 3420, request a technician visit to improve the operation of the scientific instrument 3410, order consumables or replacement parts associated with the scientific instrument 3410, or for other purposes.

[0356] Remote computing device 3440 may be a computing device located remotely from scientific instrument 3410 or user local computing device 3420 (e.g., any embodiment of the computing device 3300 discussed herein). In some embodiments, remote computing device 3440 may be included in a data center or other large-scale server environment. In some embodiments, remote computing device 3440 may include network-attached storage (e.g., as part of storage device 3404). Remote computing device 3440 may store data generated by scientific instrument 3410, perform analysis on data generated by scientific instrument 3410 (e.g., according to programming instructions), facilitate communication between user local computing device 3420 and scientific instrument 3410, or facilitate communication between service local computing device 3430 and scientific instrument 3410.

[0357] In some implementations, this can be omitted. Figure 34 One or more of the components of the illustrated scientific instrument support system 3400. Furthermore, in some embodiments, Figure 34 Multiple components of the various elements in the scientific instrument support system 3400 may exist. For example, the scientific instrument support system 3400 may include multiple user local computing devices 3420 (e.g., different user local computing devices 3420 associated with different users or located in different locations). As another example, the scientific instrument support system 3400 may include multiple scientific instruments 3410, all of which communicate with a serving local computing device 3430 and / or a remote computing device 3440; in this embodiment, the serving local computing device 3430 may monitor these multiple scientific instruments 3410, and the serving local computing device 3430 may cause updates or other information to be simultaneously “broadcast” to multiple scientific instruments 3410. The different scientific instruments 3410 in the scientific instrument support system 3400 may be close to each other (e.g., in the same room) or far from each other (e.g., on different floors of a building, in different buildings, in different cities, etc.). In some implementations, scientific instrument 3410 may be connected to an Internet of Things (IoT) stack that allows command and control of scientific instrument 3410 via web-based applications, virtual or augmented reality applications, mobile applications, or desktop applications. Any of these applications may be accessible to a user operating a user-local computing device 3420 that communicates with scientific instrument 3410 via an intermediate remote computing device 3440. In some implementations, scientific instrument 3410 may be sold by the manufacturer along with one or more associated user-local computing devices 3420 that are part of a local scientific instrument computing unit 3412.

[0358] In some embodiments, the different scientific instruments 3410 included in the scientific instrument support system 3400 can be of different types; for example, one scientific instrument 3410 may be a mass spectrometer, while another scientific instrument 3410 may be a chromatograph or an autosampler. In some such embodiments, a remote computing device 3440 or a user-local computing device 3420 may combine data from the different types of scientific instruments 3410 included in the scientific instrument support system 3400.

[0359] In various contexts, machine learning algorithms or models can be implemented in any suitable manner to facilitate any suitable aspect described herein. To facilitate some of the machine learning aspects described above in various implementations, consider the following discussion of artificial intelligence (AI). The various implementations described herein can employ artificial intelligence to facilitate the automation of one or more features or functions. These components can employ various AI-based schemes to perform the various implementations / examples disclosed herein. To provide or assist in the numerous determinations (e.g., determination, ascertainment, inference, computation, prediction, prognosis, estimation, derivation, forecasting, detection, computation) described herein, the components described herein can examine all or a subset of the data to which they have been granted access and can provide reasoning or determination of the state of a system or environment from a set of observations, such as those captured via events or data. For example, determinations can be employed to identify specific contexts or actions, or a probability distribution of states can be generated. These determinations can be probabilistic; that is, the probability distribution of states of interest is computed based on considerations of data and events. Determination can also refer to techniques employed to compose higher-level events from a collection of events or data.

[0360] This determination can lead to the construction of new events or actions from observed events or a collection of stored event data, regardless of whether the events are closely related in time or whether the events and data come from one or more event and data sources. The components disclosed herein can employ various classification schemes (explicit training (e.g., via training data) and implicit training (e.g., via observed behavior, preferences, historical information, received external information, etc.)) or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, etc.) related to the execution of automatic or deterministic actions relevant to the claimed subject matter. Therefore, classification schemes or systems can be used to automatically learn and execute multiple functions, actions, or determinations.

[0361] The classifier can take the input attribute vector z = (z1, z2, z3, z4, z5) as input. nThe classification maps the input to a confidence level that it belongs to a certain class, such as f(z) = confidence level (class). This classification can use probabilistic or statistical analysis (e.g., considering the utility and cost of the analysis) to determine the action to be automatically performed. Support Vector Machines (SVMs) can be used as an example of a classifier that can be adopted. SVMs operate by finding a hypersurface in the space of possible inputs, where the hypersurface attempts to separate triggering criteria from non-triggering events. Intuitively, this makes the classification correct for test data that is close to but not identical to the training data. Other directed and non-directed model classification methods include, for example, Naive Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models, or probabilistic classification models that provide different independent patterns, any of which can be adopted. The classification used in this paper also includes statistical regression for developing priority models.

[0362] To provide additional context for the various implementation schemes described herein, Figure 35 The following discussion is intended to provide a brief, general description of a suitable computing environment 3500 for various implementations of the embodiments described herein. While the embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that the embodiments may also be implemented in combination with other program modules or as a combination of hardware and software.

[0363] Typically, program modules include routines, programs, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will understand that the methods of this invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each operatively coupled to one or more associated devices.

[0364] The implementation schemes shown in this paper can also be practiced in distributed computing environments, where some tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside on both local and remote memory storage devices.

[0365] Computing devices typically include a variety of media, which may include computer-readable storage media, machine-readable storage media, or communication media. These two terms are used interchangeably herein, as follows. A computer-readable storage media or a machine-readable storage media can be any available storage medium that can be accessed by a computer, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, a computer-readable storage media or a machine-readable storage media can be implemented in conjunction with any method or technology used for storing information, such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0366] Computer-readable storage media may include, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compressed optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), Blu-ray disc (BD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible or non-transitory media that can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” used herein to describe storage devices, memories, or computer-readable media should be understood to exclude only the propagation of transient signals themselves as a modifier, and do not waive the rights of all standard storage devices, memories, or computer-readable media that do not merely propagate transient signals themselves.

[0367] Computer-readable storage media can be accessed by one or more local or remote computing devices, for example via access requests, queries or other data retrieval protocols, for various operations concerning the information stored on the media.

[0368] Communication media typically contain computer-readable instructions, data structures, program modules, or other structured or unstructured data in data signals (such as modulated data signals, such as carrier waves or other transmission mechanisms), and include any information transmission or delivery medium. The term "modulated data signal" or signal refers to a signal whose characteristics are set or altered to encode information in one or more signals. By way of example, and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media).

[0369] Refer again Figure 35Example environment 3500 for implementing various embodiments of the aspects described herein includes a computer 3502, which includes a processing unit 3504, system memory 3506, and a system bus 3508. The system bus 3508 couples system components, including but not limited to system memory 3506, to the processing unit 3504. The processing unit 3504 can be any of a variety of commercially available processors. Dual microprocessors and other multiprocessor architectures may also be used as the processing unit 3504.

[0370] System bus 3508 can be any of several types of bus architectures, and can further interconnect with memory buses (with or without memory controllers), peripheral buses, and local buses u...

Claims

1. A system comprising: A processor that executes a computer-executable component stored in a non-transitory computer-readable storage memory, wherein the computer-executable component includes: An access component that accesses natural language instructions associated with a charged particle microscope, wherein the natural language instructions request or command the charged particle microscope to undergo configurable setting adjustments or to perform automated tasks. A state component, in response to the receipt of the natural language instruction, causes the charged particle microscope to capture an image or energy spectrum of the sample currently mounted on the stage of the charged particle microscope according to a default microscopy protocol; and A model component performs a large language model on both the natural language instruction and the image or energy spectrum of the sample, thereby generating a natural language response indicating how implementing the natural language instruction will affect the sample.

2. The system of claim 1, wherein the computer-executable component further comprises: The renderer component, the renderer component: The natural language response or a visual graphic associated with the natural language response is visually presented on an electronic display associated with the charged particle microscope; The natural language response is audibly played on an electronic speaker associated with the charged particle microscope; or The natural language response is transmitted to a computing device associated with the charged particle microscope.

3. The system of claim 1, wherein the natural language instruction is: plain text typed into a text field of a graphical user interface associated with the charged particle microscope; or plain text transcribed from an audio recording captured by a microphone associated with the charged particle microscope.

4. The system of claim 1, wherein the natural language response indicates that implementing the natural language instruction will damage the sample or charge the sample, and wherein the natural language response further indicates that the charged particle microscope should undergo alternative configurable setting adjustments or that the charged particle microscope should perform alternative automated tasks.

5. The system of claim 4, wherein the computer-executable component further comprises: A presenter assembly that enables the charged particle microscope to undergo the alternative configurable settings adjustment or perform the alternative automated tasks.

6. The system of claim 1, wherein the computer-executable component further comprises: A context component identifies one or more documents related to the natural language instruction and the image or energy spectrum of the sample via an embedding search of a document repository, wherein the large language model receives the natural language instruction, the image or energy spectrum of the sample, and the one or more documents as input.

7. The system of claim 1, wherein the computer-executable component further comprises: A context component that performs one or more available deep learning models on the image or energy spectrum of the sample to produce one or more inference task results, wherein the large language model receives the natural language instructions, the image or energy spectrum of the sample, and the one or more inference task results as input.

8. The system of claim 1, wherein the charged particle microscope is synchronized with the digital twin, and wherein: The large language model receives the natural language instructions, the image or energy spectrum of the sample, and the current health status of the charged particle microscope as indicated by the digital twin as input; or The large language model receives the natural language instructions, the image or energy spectrum of the sample, and one or more simulation results about the charged particle microscope generated by the digital twin as input.

9. A computer-implemented method, the computer-implemented method comprising: Devices operatively coupled to the processor access natural language instructions associated with the charged particle microscope, wherein the natural language instructions request or command the charged particle microscope to undergo configurable setting adjustments or perform automated tasks. The device, in response to the receipt of the natural language command, causes the charged particle microscope to capture an image or energy spectrum of the sample currently mounted on the stage of the charged particle microscope according to a default microscopy protocol; and The device performs a large language model on both the natural language instruction and the image or energy spectrum of the sample, thereby generating a natural language response indicating how implementing the natural language instruction will affect the sample.

10. The computer-implemented method according to claim 9, further comprising: The device visually presents the natural language response or a visual graphic associated with the natural language response on an electronic display associated with the charged particle microscope; The natural language response is audibly played by the device on an electronic speaker associated with the charged particle microscope; or The device transmits the natural language response to a computing device associated with the charged particle microscope.

11. The computer-implemented method of claim 9, wherein the natural language instruction is: plain text typed into a text field of a graphical user interface associated with the charged particle microscope; or plain text transcribed from an audio recording captured by a microphone associated with the charged particle microscope.

12. The computer-implemented method of claim 9, wherein the natural language response indicates that implementing the natural language instruction will damage the sample or charge the sample, and wherein the natural language response further indicates that the charged particle microscope should undergo alternative configurable setting adjustments or that the charged particle microscope should perform alternative automated tasks.

13. The computer-implemented method according to claim 12, further comprising: The device enables the charged particle microscope to undergo the alternative configurable settings adjustment or perform the alternative automated tasks.

14. The computer-implemented method according to claim 9, further comprising: One or more documents related to the natural language instruction and the image or energy spectrum of the sample are identified by the device and via an embedding search of a document repository, wherein the large language model receives the natural language instruction, the image or energy spectrum of the sample, and the one or more documents as input.

15. The computer-implemented method according to claim 9, further comprising: The device executes one or more available deep learning models on the image or energy spectrum of the sample to produce one or more inference task results, wherein the large language model receives the natural language instructions, the image or energy spectrum of the sample, and the one or more inference task results as input.

16. The computer-implemented method of claim 9, wherein the charged particle microscope is synchronized with the digital twin, and wherein: The large language model receives the natural language instructions, the image or energy spectrum of the sample, and the current health status of the charged particle microscope as indicated by the digital twin as input; or The large language model receives the natural language instructions, the image or energy spectrum of the sample, and one or more simulation results about the charged particle microscope generated by the digital twin as input.

17. A computer program product for facilitating large language model assistance in charged particle microscopy operations, the computer program product comprising a non-transitory computer-readable storage memory containing program instructions executable by a processor to cause the processor to: Access a plain text command provided by the user of the scanning electron microscope, wherein the plain text command requests the scanning electron microscope to perform a specified microscopy action; In response to the receipt of the plain text command, the scanning electron microscope captures an image or energy spectrum of the sample currently mounted on the stage of the scanning electron microscope via a default microscopy protocol. A large language model is performed on both the plain text command and the image or energy spectrum of the sample, wherein the large language model produces a plain text response as output, the plain text response indicating whether the specified microscopy action will damage the sample; and The plain text response is presented visually or audibly on an electronic display or electronic speaker associated with the scanning electron microscope.

18. The computer program product of claim 17, wherein the plain text response indicates that the specified microscopy action will damage the sample, and wherein the plain text response further indicates that such damage can be avoided by alternative microscopy actions.

19. The computer program product of claim 18, wherein the program instructions are further executable to cause the processor to: The scanning electron microscope is instructed to perform the alternative microscopy procedure.

20. The computer program product of claim 17, wherein the large language model receives the plain text command, the image or energy spectrum of the sample, and one or more simulation results generated by a digital twin synchronized with the scanning electron microscope as input.

Images