Solving large-scale language model disruptions in scientific instrument workflows

Abstract

This system / technique provides a way to facilitate the resolution of large-scale language models of disruptions in scientific instrument workflows. [Solution] In various embodiments, the system can cause scientific instruments (e.g., charged particle microscopes, chromatographs, mass spectrometers) to perform a workflow on a sample. In various embodiments, the system can retrieve runtime data logged by the scientific instrument during the workflow in response to the scientific instrument generating an error message that interrupts the workflow. In various cases, the system can synthesize a first text explaining why the workflow was interrupted by running a large-scale language model on the error message and runtime data.

Description

Technical Field

[0001] The present invention relates to a large language model solution for scientific instrument workflow interruptions.

Background Art

[0002] The technical field of scientific instruments has historically been restricted by the complexity of its operations. Such operational complexity can prevent efficient troubleshooting of errors that occur in scientific instruments.

Summary of the Invention

[0003] The following presents an overview for providing a basic understanding of one or more embodiments. This overview is not intended to identify key or important elements or to delineate any scope of particular embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to a more detailed description that will be presented later. In one or more embodiments described herein, a device, system, computer-implemented method, apparatus, or computer program product that facilitates a large language model solution for scientific instrument workflow interruptions is described.

[0004] A system is provided according to one or more embodiments. The system may include a non-temporary computer-readable memory capable of storing computer-executable components. The system may further include a processor that can be operably coupled to the non-temporary computer-readable memory and can execute the computer-executable components stored in the non-temporary computer-readable memory. In various embodiments, the computer-executable component may include a workflow component that causes a charged particle microscope to perform a workflow on a sample. In various aspects, the computer-executable component may include a status component that can retrieve runtime data logged by the charged particle microscope during the workflow in response to the charged particle microscope generating an error message that interrupts the workflow. In various cases, the computer-executable component may include a model component that can synthesize a first text explaining why the workflow was interrupted, through the execution of a large-scale language model on the error message and runtime data.

[0005] According to one or more embodiments, a computer-aided implementation method is provided. In various embodiments, the computer-aided implementation method may include a device operably coupled to a processor causing a charged particle microscope to perform a workflow on a sample. In various aspects, the computer-aided implementation method may include the device retrieving runtime data logged by the charged particle microscope during the workflow, in response to the charged particle microscope generating an error message that interrupts the workflow. In various cases, the computer-aided implementation method may include the device synthesizing a first text explaining why the workflow was interrupted, through the execution of a large-scale language model on the error message and runtime data.

[0006] According to one or more embodiments, a computer program product is provided for facilitating the resolution of large-scale language models of scientific instrument workflow interruptions. In various embodiments, the computer program product may include non-temporary computer-readable memory, which has program instructions embodied in the non-temporary computer-readable memory. In various embodiments, the program instructions may be executable by a processor by causing the program to cause the scientific instrument to perform a workflow on a sample. In various embodiments, the program instructions are executable by a processor and cause the processor to retrieve runtime data logged by the scientific instrument during the workflow in response to the scientific instrument generating an error message that interrupts the workflow. In various embodiments, the program instructions may be executable by a processor by causing the processor to synthesize a first text explaining why the workflow was interrupted, through the execution of a large-scale language model on the error message and runtime data. In various embodiments, the scientific instrument may be a charged particle microscope, a chromatograph, or a mass spectrometer. [Brief explanation of the drawing]

[0007] Various embodiments will be readily apparent from the following detailed description in conjunction with the accompanying drawings. For the sake of this specification, similar reference numerals indicate similar structural elements. Embodiments are illustrated in the drawings as examples, not as limitations. The drawings are not necessarily drawn to scale. [Figure 1] This specification illustrates exemplary and non-limiting block diagrams of scientific instrument modules according to various embodiments described herein. [Figure 2] This specification illustrates illustrative and non-limiting flowcharts of computer implementations according to various embodiments described herein. [Figure 3] This specification illustrates an exemplary, non-limiting block diagram of a system that facilitates large-scale language model resolution of scientific instrument workflow interruptions, based on one or more embodiments described herein. [Figure 4] This specification illustrates an exemplary, non-limiting system block diagram, including an instrument workflow and error messages, that facilitates large-scale language model resolution of scientific instrument workflow interruptions, based on one or more embodiments described herein. [Figure 5] This specification illustrates an exemplary, non-limiting block diagram showing how an error message can interrupt the equipment workflow in one or more embodiments described herein. [Figure 6] This specification illustrates an exemplary, non-limiting block diagram of a system, including instrument runtime data, that facilitates large-scale language model resolution of scientific instrument workflow interruptions, based on one or more embodiments described herein. [Figure 7] This specification illustrates exemplary and non-limiting block diagrams of device runtime data according to one or more embodiments described herein. [Figure 8] This specification illustrates an exemplary, non-limiting block diagram of a system including a technical documentation repository and solutions that facilitates large-scale language model resolution of scientific instrument workflow interruptions, according to one or more embodiments described herein. [Figure 9] This specification illustrates an exemplary, non-limiting block diagram showing how solutions to error messages can be generated based on device runtime data and a technical documentation repository according to one or more embodiments described herein. [Figure 10] This specification illustrates an exemplary, non-limiting block diagram showing how solutions to error messages can be generated based on device runtime data and a technical documentation repository according to one or more embodiments described herein. [Figure 11] This specification illustrates an exemplary, non-limiting block diagram showing how solutions to error messages can be generated based on device runtime data and a technical documentation repository according to one or more embodiments described herein. [Figure 12]This specification illustrates an exemplary, non-limiting block diagram showing how solutions to error messages can be generated based on device runtime data and a technical documentation repository according to one or more embodiments described herein. [Figure 13] This specification illustrates an exemplary, non-limiting block diagram showing how solutions to error messages can be generated based on device runtime data and a technical documentation repository according to one or more embodiments described herein. [Figure 14] This specification illustrates an exemplary, non-limiting block diagram showing how solutions to error messages can be generated based on device runtime data and a technical documentation repository according to one or more embodiments described herein. [Figure 15] This specification illustrates an exemplary, non-limiting block diagram of a system, including solution feedback and a set of technical documentation editors, that facilitates large-scale language model resolution of scientific instrument workflow interruptions, according to one or more embodiments described herein. [Figure 16] This specification illustrates an exemplary, non-limiting block diagram showing how one or more technical document edits can be generated based on solution feedback according to one or more embodiments described herein. [Figure 17] This specification illustrates an exemplary, non-limiting block diagram showing how large-scale language models can be trained using one or more embodiments described herein. [Figure 18] This specification illustrates an exemplary and non-limiting block diagram of an operating environment that may facilitate one or more embodiments described herein. [Figure 19] This specification illustrates exemplary networking environments capable of operating to perform the various implementations described herein. [Figure 20] This specification illustrates an exemplary dual-beam microscope that can be implemented according to various embodiments described herein. [Modes for carrying out the invention]

[0008] The following detailed description is merely illustrative and is not intended to limit the application or use of the embodiments. Further, no constraint is intended by any of the expressions or implied information presented in the preceding "Background Art" or "Summary of the Invention" sections, or in the "Mode for Carrying Out the Invention" section.

[0009] Here, one or more embodiments will be described with reference to the drawings, and like reference numerals will be used throughout the drawings to refer to like elements. In the following description, for the sake of convenience of explanation, a number of specific details will be set forth in order to provide a more detailed understanding of one or more embodiments. However, it is clear that in various cases, one or more embodiments can be implemented without these specific details.

[0010] Various operations can be described in sequence as a plurality of separate actions or operations in a manner that is most helpful for understanding the subject matter disclosed herein. However, the order of description should not be construed as suggesting that these operations necessarily depend on the order. In particular, these operations can be performed in an order different from the presented order. The operations described can be performed in an order different from the described embodiments. In additional embodiments, various additional operations can be performed or the described operations can be omitted.

[0011] Some elements may be referred to in the singular (e.g., "processing device"), but any suitable element can be represented by a plurality of instances of that element and vice versa. For example, a set of operations described as being performed by a processing device can be implemented using different ones of the operations performed by different processing devices. As used herein, the phrase "based on" should be understood to mean "at least partially based on" unless otherwise specified.

[0012] Scientific instruments are a highly complex combination of hardware and software, utilizing a complex configuration of operable components (e.g., ion sources, electron sources, optical lenses or apertures, optical plates or deflectors, columns, coils, heaters, coolers, fluid valves, fluid pumps, circuit switches, sample stages), sensors (e.g., ion detectors, electron detectors, voltmeters, thermistors, potentiometers, pressure gauges), or consumables (e.g., carrier fluids, calibration materials, filters, reactive gases) to facilitate high-precision measurement of physical samples (e.g., semiconductor wafers, lamellae, aqueous mixtures, biological tissues) in scientific, laboratory, research, or clinical operating environments. For example, charged particle microscopes (e.g., scanning electron microscopes (SEMs), transmission electron microscopes) electron microscope (TEM), electron energy loss microscope (electron An energy-loss microscope (EELM) is a type of scientific instrument capable of capturing or generating microscopic or nanoscale images or energy spectra in scientific, laboratory, research, or clinical operating environments. Another example is a mass spectrometer, which can be coupled to or equipped with a chromatograph, is another type of scientific instrument capable of capturing or generating ion abundance data (e.g., chromatograms, mass spectra) associated with the chemical or molecular composition of a physical sample.

[0013] The technical field of scientific instrumentation has historically been constrained by the complexity of its operation. In other words, scientific instruments can have such complex structures that their operation or use can accordingly become complex. In fact, in order for a user to use a scientific instrument competently or confidently to analyze clinical or laboratory samples, that user often requires extensive specialized training, education, or certification regarding that scientific instrument. For example, a user can learn how to appropriately operate the graphical user-interface (GUI) or physical controls of a charged particle microscope or a mass spectrometer by taking an instrumentation course that is several weeks or months in length. Ultimately, a charged particle microscope or a mass spectrometer can, at first glance, have a large number or overwhelming number of configurable software or hardware settings, buttons, knobs, sliders, or options. A user who has not experienced such extensive learning may not be able to fully use a charged particle microscope or a mass spectrometer. For example, a user who attempts to operate a charged particle microscope without taking an appropriate instrumentation course that is several weeks or months in length may be significantly at risk of damaging the sample or the instrument itself.

[0014] The need for extensive learning or skills can be a significant problem, given that different types of charged particle microscopes can function or be operated differently from one another. That is, specialized instrumentation training, education, or certification is not transferable between different types of scientific instruments, within different scientific instruments, or between different scientific instruments. For example, any training that prepares a person to operate a SEM competently does not necessarily prepare them to operate a TEM competently, nor does it prepare them to operate a mass spectrometer competently. As another example, any training that prepares a person to operate an SEM with model number A competently does not necessarily prepare them to operate an SEM with model number B competently. As yet another example, any training that prepares a person to operate an SEM with model number A and software version C competently does not necessarily prepare them to operate an SEM with model number A and software version D competently.

[0015] Because the operation is so complex, users of scientific instruments often experience and are unable to resolve hardware or software malfunctions that occur during the operation of such instruments. To troubleshoot such malfunctions or to address them alternatively, manufacturers of scientific instruments often provide users with technical support services. For example, a manufacturer may interact with a user using a large number of field engineers, technical experts, or R&D scientists with the goal or purpose of troubleshooting any instrumentation malfunction the user may encounter. Depending on the severity or extent of any given malfunction (for example, the given malfunction may be a simple, commonly occurring problem, or it may instead be an unprecedented software glitch or unprecedented hardware failure), these technical support services may consume an excessive amount of time (e.g., hours, days, or even weeks) to evaluate the given malfunction, brainstorm solutions to the given malfunction, and actually implement or run those solutions to resolve the given malfunction. These problems can increase exponentially as the number of scientific instruments supplied by manufacturers to different users increases.

[0016] Therefore, a system or technology that can reduce the time required to troubleshoot or resolve malfunctions that may occur during the operation of scientific instruments may be desirable.

[0017] Various embodiments described herein can address this technical problem. One or more embodiments described herein may include a system, computer implementation, apparatus, or computer program product that can facilitate large-scale language model resolution of scientific instrument workflow interruptions. In other words, various embodiments described herein can leverage large language models (LLMs) such as ChatGPT to bring about rapid and effective troubleshooting of scientific instrument malfunctions. In fact, when various embodiments described herein are implemented, errors, failures, or other malfunctions that interrupt scientific instrument workflows can be resolved in just seconds or minutes, rather than hours, days, or weeks. Thus, users of such scientific instruments do not need to wait excessively long for troubleshooting services or repairs to be provided, brainstormed, or developed by field engineers, technical experts, or scientific researchers. Instead, an LLM can automatically provide, brainstorm, or develop such troubleshooting services or repairs in real time, as described herein, which may be considered desirable, beneficial, or advantageous.

[0018] The inventors have devised various embodiments to reduce the amount of time consumed through scientific instrument troubleshooting. As described herein, such reduction in time consumption can be achieved by implementing malfunction-triggered LLM solution synthesis and feedback-conditional LLM editing of solution references.

[0019] Firstly, consider malfunction-triggered LLM resolution synthesis. In various embodiments, a scientific instrument may be loaded with a given sample and may initiate or begin executing a given workflow (e.g., imaging workflow, injection workflow, milling workflow, spatial repositioning workflow, thermal adjustment workflow) on that given sample. In various embodiments, a given workflow may be interrupted by a specific malfunction. For example, a scientific instrument may throw or present one or more error codes that abort or pause the workflow. While it may be desirable to quickly diagnose, troubleshoot, or resolve a specific malfunction, obtaining such diagnosis, troubleshooting, or resolution from a field engineer, technical expert, or scientific researcher associated with the scientific instrument manufacturer can be very time-consuming (e.g., it may take several weeks, depending on how complex the particular malfunction is). Therefore, in various aspects, a scientific instrument can respond to a particular malfunction by extracting or recalling its runtime data associated with a given workflow (for example, by extracting or recalling any image or mass spectrum partially captured by the scientific instrument during a given workflow, by extracting or recalling any values ​​set for the controllable operating parameters of the scientific instrument during a given workflow, or by extracting or recalling any other data captured by any other sensors of the scientific instrument during a given workflow). In various cases, the runtime data and error codes input by the scientific instrument are fed together to the LLM as a collective input prompt, thereby causing the LLM to create various fragments of synthesized text content relating to a particular malfunction. Some of the synthesized text content may be one or more natural language sentences describing or explaining what is believed to have caused the particular malfunction in the scientific instrument. Other parts of the synthesized text content may be one or more natural language sentences describing or explaining how to resolve, repair, or correct the particular malfunction.Further portions of the synthesized text content may be one or more lines of computer executable code that can actually implement at runtime any software-related steps, actions, or modifications that resolve, repair, or correct a particular malfunction. In other words, error codes and runtime data generated by a scientific instrument can be considered to convey at least a certain amount of measured or quantifiable information unique or specific to a particular malfunction (in some cases in a hidden, uninterpretable, or otherwise obscured manner), and the LLM can be considered to leverage such information to predict or infer how to resolve a particular malfunction. In some embodiments, error codes and runtime data may be augmented in a retrieval-augmented generative (RAG) manner by relevant technical documentation associated with the scientific instrument (e.g., by the scientific instrument's operating manual, by the scientific instrument's design blueprint, or by coding scripts or files associated with the scientific instrument). In either case, the synthesized text content can be considered to instruct or explain why a particular malfunction is occurring or how to prevent it from occurring, and the LLM can generate such synthesized text content in as little as a few seconds. Therefore, triggering the LLM to synthesize a troubleshooting solution for a particular malfunction in response to the malfunction disrupting a given workflow, as described herein, can help ensure that the malfunction is resolved quickly. This should be contrasted with the fact that it typically takes field engineers, technical experts, or scientific researchers days or weeks to attempt to resolve complex malfunctions occurring in their scientific instruments.

[0020] Next, consider LLM editing with feedback based on resolution references. As mentioned above, in response to a given workflow being interrupted by a particular malfunction, LLM can generate synthesized text content that explains or describes (in LLM's opinion) why the particular malfunction occurred or (again, in LLM's opinion) how to resolve the particular malfunction. Also, as mentioned above, such synthesized text content may be based on one or more relevant technical documents associated with scientific instruments and treated as RAG references in LLM. Here, in various aspects, a field engineer, technical expert, or scientific researcher may provide feedback on the synthesized text content. In various cases, this feedback may be natural language text written by the field engineer, technical expert, or scientific researcher criticizing or otherwise identifying shortcomings or inaccuracies in the synthesized text content. In response, this feedback, the synthesized text content, and the RAG references on which LLM relied to generate the synthesized text content are all fed together to LLM as a collective input prompt, thereby allowing LLM to create additional fragments of synthesized text content. In various cases, these additional fragments of synthesized text content may consist of one or more edits (e.g., word deletion, word insertion) that, if incorporated into each of the RAG references before the generation of the synthesized text content, would result in the synthesized text content being missing or having defects or inaccuracies specified in the feedback. In other words, the feedback can be seen as precisely indicating specific inaccuracies in the LLM's synthesized troubleshooting solution for a particular malfunction, and the LLM can leverage such feedback to determine which parts of which RAG references caused those specific inaccuracies in the LLM and synthesize edits to those parts of those RAG references to help the LLM prevent those same specific inaccuracies from occurring again in the future.Whatever shortcomings in the solution identified in the feedback, allowing the LLM to infer how to edit the RAG reference to avoid them increases the likelihood that the LLM will provide accurate solution inferences when responding to future device malfunctions. In some cases, this may be referred to as reverse RAG (for example, RAG may include synthesizing text content using relevant text references). In contrast, reverse RAG may include using feedback on the synthesized text content to edit the relevant text references.

[0021] The various embodiments described herein can be considered as computerized tools (e.g., any preferred combination of computer executable hardware or computer executable software) that can facilitate the resolution of large-scale language models of scientific instrument workflow interruptions. In various embodiments, such computerized tools may include workflow components, status components, model components, or execution components.

[0022] In various embodiments, scientific instruments may exist. In some cases, a scientific instrument may be a charged particle microscope exhibiting any preferred design or structure (e.g., it may be a SEM, TEM, EELM, or dual-beam microscope). In other cases, a scientific instrument may be a chromatograph-equipped mass spectrometer exhibiting any preferred design or structure (e.g., liquid chromatography hardware, gas chromatography hardware, ion chromatography hardware, matrix-assisted laser desorption / ionization (MALDI) hardware, electrospray ionization (ESI) hardware, quadrupole mass filter analyzer hardware, ion trap analyzer hardware, time-of-flight (TOF) analyzer hardware, or electrostatic trap hardware). In various cases, a scientific instrument may include any number of preferred configurable operating settings. In various embodiments, the configurable operating settings may be any suitable selectively controllable hardware or software characteristics of the scientific instrument that can be directly adjusted or modified in response to electronic instructions or commands received from the user of the scientific instrument (e.g., user-controlled voltage or current settings of the scientific instrument, user-controlled temperature settings of the scientific instrument, or user-controlled actuator settings of the scientific instrument). In various embodiments, there may be any suitable sample currently loaded into the scientific instrument (e.g., currently located or positioned on the operable stage of a charged particle microscope, currently inside the autosampling injector of a chromatograph-equipped mass spectrometer) (e.g., a semiconductor wafer or lamella for a charged particle microscope, an aqueous mixture for a mass spectrometer).

[0023] In various embodiments, LLMs can exist. In various aspects, an LLM can exhibit any suitable deep learning internal architecture. For example, an LLM can include any suitable number of any suitable types of layers (e.g., an input layer, one or more hidden layers, and an output layer, any of which may be a convolutional layer, a high-density layer, a long short-term memory (LSTM) layer, a transformer layer, a nonlinear layer, a pooling layer, a batch normalization layer, or a padding layer). Another example is that an LLM can include any suitable number of neurons in various layers (e.g., different layers may have the same or different numbers of neurons). Yet another example is that an LLM can include any suitable activation function in various neurons (e.g., softmax, sigmoid, hyperbolic tangent, normalized linear unit) (e.g., different neurons may have the same or different activation functions). Yet another example is that an LLM can include any suitable inter-neuron or inter-layer connections (e.g., forward connections, skip connections, recursive connections).

[0024] Regardless of the specific internal architecture of LLM, LLM can be configured as a generative text-to-text model. That is, LLM can be configured to receive any suitable text data (which may or may not be accompanied by any suitable numerical data or any suitable graphical data in various cases) as input, and to generate synthesized text content (e.g., one or more synthesized sentences or sentence fragments) as output, based semantically or substantively on such input text data (and possibly accompanying numerical or graphical data).

[0025] To accomplish this, the LLM can be considered to include an encoder portion and a synthesizer portion. In various embodiments, the encoder portion may be any preferred upstream layer of the LLM configured to receive input text data (and any accompanying numerical or graphical data) and to generate embeddings based on the input text data. In various embodiments, the synthesizer portion may be any preferred downstream layer of the LLM configured to receive those embeddings and to generate synthesized text content based on those embeddings.

[0026] In various aspects, the embeddings generated by the encoder portion of the LLM in response to a portion of the input text, numerical, or graphical data can be considered as any suitable mathematical quantity (e.g., scalar, vector, matrix, tensor, tokenization, or any suitable combination thereof) that numerically represents at least some substantial or semantic aspect of the input text, numerical, or graphical data in a low-dimensional manner. In other words, the embeddings can be made smaller in terms of size or dimensionality than such input text, numerical, or graphical data (e.g., by more than an order of magnitude in some cases), but despite such smaller size, the embeddings can be considered to represent such input text, numerical, or graphical data substantially or semantically. In other words, the embeddings can be considered as a latent vector representation of such input text, numerical, or graphical data.

[0027] In any case, it may be desirable to utilize LLM to troubleshoot or resolve malfunctions occurring in scientific instruments. In various cases, the computerized tools described herein can accomplish this.

[0028] In various embodiments, a computerized tool can electronically access an LLM or scientific instrument. For example, a computerized tool can electronically interface with or communicate with an LLM or scientific instrument (e.g., send electronic commands to an LLM or scientific instrument, read electronic signals from an LLM or scientific instrument). Thus, any component of a computerized tool can electronically interact with an LLM or scientific instrument (e.g., read, write, edit, copy, manipulate, execute, start, stop, modify).

[0029] In various embodiments, a computerized tool's workflow component can electronically initiate an instrument workflow on a scientific instrument, for whatever sample is currently loaded into the instrument. In various embodiments, the instrument workflow may be any preferred sequence of one or more hardware or software operations that the scientific instrument can perform. For example, suppose the scientific instrument is a charged particle microscope. In such a case, the instrument workflow may be an imaging protocol configured or intended to capture a microscopic or nanoscale image of the sample, a milling protocol configured or intended to grind or remove a certain amount of material from a specific location on the sample, a stage operation protocol configured or intended to physically move or reorient the sample in space, or a vacuum protocol configured or intended to pump or ventilate the vacuum chamber in which the sample is located to a specific pressure level. As another example, suppose the scientific instrument is a chromatograph-equipped mass spectrometer. In such cases, the instrument workflow may be a scanning or screening protocol configured or intended to capture a chromatogram or mass spectrum of a sample, a flushing protocol configured to flush various channels or conduits of the mass spectrometer with the sample (when the sample is a washing mixture), or a temperature control protocol configured or intended to raise or lower the temperature of various components of the mass spectrometer (e.g., oven heating column, autosampling injector) to a specified temperature level. In some cases, the instrument workflow may be indicated or provided by the user of the scientific instrument. For example, the user may notify the scientific instrument of the instrument workflow by interacting with any suitable human-computer interface of the scientific instrument (e.g., touchscreen, keyboard, voice control system). In any case, the scientific instrument may initiate or attempt to perform the instrument workflow on the sample.

[0030] In various forms, instrument workflows can be interrupted by error messages. That is, a scientific instrument may abruptly or suddenly abort or pause an instrument workflow and generate an error message while attempting to perform whatever sequence of hardware or software operations constitutes the instrument workflow. In other words, an instrument workflow may consist of a series of operations performed sequentially, and the scientific instrument may, after performing whatever the first or initial operation included in the instrument workflow, but before performing whatever the last or final operation included in the instrument workflow, abort the progress of such operations and instead display an error message. In any case, the error message may be any suitable electronic data that (explicitly or implicitly) indicates that the scientific instrument is malfunctioning in some way and is unable to complete or terminate the instrument workflow. For example, an error message may be structured text (e.g., an alphanumeric identifier) ​​that somehow corresponds to or is associated with the occurrence of the malfunction, unstructured text (e.g., one or more natural language sentences or sentence fragments) that semantically describes or states the occurrence of the malfunction or any other details, or visual symbols (e.g., a forbidden / crossed circle symbol, a frown symbol) that visually indicate the occurrence of the malfunction or any other details.

[0031] In various embodiments, the status component of a computerized tool may, in response to the generation of an error message, cause the scientific instrument to access or electronically collect instrument runtime data. In various embodiments, instrument runtime data may be any suitable electronic data generated, logged, used, or otherwise tracked by the scientific instrument during its partial or partial execution of the instrument workflow. As some examples, runtime data may include any values ​​or states of the configurable operating parameters of the scientific instrument during its partial or partial execution of the instrument workflow, any imaging or spectroscopic measurement data partially captured by the scientific instrument during its partial or partial execution of the instrument workflow, or measurements recorded during its partial or partial execution of the instrument workflow by auxiliary or incidental sensors (e.g., pressure sensors, video cameras, microphones, temperature sensors, humidity sensors) incorporated into or built into the scientific instrument. Thus, runtime data can be considered to communicate the operating state of the scientific instrument up to and at the moment the instrument workflow is interrupted by a malfunction.

[0032] In various embodiments, the model component of a computerized tool can electronically generate solutions to malfunctions by performing LLM on error messages and device runtime data.

[0033] More specifically, there may be causal prompts, which may be unstructured or plain text, that request or instruct the scientific instrument to determine what caused it to generate an error message. In various embodiments, a model component can concatenate the error message, instrument runtime data, and causal prompt together. In various cases, the model component can feed its concatenation into the input layer of an LLM, which can complete a forward pass through one or more hidden layers of the LLM, and the output layer of the LLM can compute a first natural language response based on the activations provided by one or more hidden layers of the LLM.

[0034] In various cases, the first natural language response may be based on error messages and instrument runtime data, or it may be synthesized text that responds substantively or semantically to causal prompts. In other words, the first natural language response may be unstructured or plain text that describes or explains what caused the scientific instrument to interrupt the instrument workflow (as inferred or predicted by the LLM). To put it another way, the first natural language response may textually state what hardware-related or software-related failure of the scientific instrument the LLM considers to be the root cause of the error message. That is, the error messages and instrument runtime data can be considered to provide the LLM with a snapshot of the internal operation of the scientific instrument while attempting to perform the instrument workflow, and the LLM can use that snapshot to infer or predict which particular part of the hardware or software of the scientific instrument failed and thereby interrupted the instrument workflow. In other words, the LLM can leverage any discrepancies, inconsistencies, or other interrelationships between the internal operation that the scientific instrument is expected to have during the execution of the instrument workflow and the internal operation that the scientific instrument actually had during the instrument workflow in order to infer or predict why the instrument workflow was interrupted. For example, if the instrument workflow is of a particular type of imaging protocol or imaging scan, the LLM may expect the instrument runtime data to appear or manifest in a particular manner (e.g., having or lacking a particular visual, auditory, or numerical pattern or artifact). The extent or manner to which the instrument runtime data deviates from such expectations may be considered to indicate, at least partially, or suggest to the LLM what is wrong with the scientific instrument.

[0035] In some cases, a resolution prompt may exist, which may be unstructured or plain text requesting or instructing how to resolve or correct whatever part of the hardware or software specified in the first natural language response may be. In various embodiments, a model component may concatenate an error message, device runtime data, the first natural language response, and the resolution prompt together. In various embodiments, the model component may feed its concatenation to the input layer of an LLM, which may complete a forward pass through one or more hidden layers of the LLM, and the output layer of the LLM may compute a second natural language response based on the activations provided by one or more hidden layers of the LLM.

[0036] In various cases, the second natural language response may be an error message, instrument runtime data, and synthesized text that is based on or substantially or semantically responds to the first natural language response or resolution prompt. In other words, the second natural language response may be unstructured or plain text that describes or explains to the scientific instrument what caused the instrument workflow to be interrupted and what hardware or software adjustments (inferred or predicted by the LLM) will correct. In other words, the second natural language response may textually describe what sequence of actions will correct or repair what hardware or software failures indicated in the first natural language response, if maintenance, repair, or development actions are performed on the scientific instrument. That is, the error message and instrument runtime data can be considered to provide the LLM with a snapshot of the internal operation of the scientific instrument while attempting to perform the instrument workflow, and the first natural language response can be considered to indicate a specific hardware or software failure that is inferred to have interrupted the instrument workflow, and the LLM can use such information to infer or predict what specific repair steps should be taken to repair such hardware or software failures. Ultimately, different types of equipment workflows can cause each other to fail with their respective types of hardware or software, and these failures may require their own set of repair steps.

[0037] In some cases, a code prompt may exist, which may be unstructured or plain text requesting or instructing the generation of a coding script that its execution will implement, whatever software action or software step is specified in the second natural language response. In various embodiments, a model component may concatenate an error message, instrument runtime data, the second natural language response, and the code prompt together. In various embodiments, the model component may feed its concatenation to the input layer of the LLM, which may complete a forward pass through one or more hidden layers of the LLM, and the output layer of the LLM may compute the coding script based on the activations provided by one or more hidden layers of the LLM.

[0038] In various cases, a coding script may be written in any preferred programming syntax and may substantially or functionally satisfy a code prompt. In other words, a coding script may be a synthesized line of computer executable code configured to perform or implement, at runtime, any corrective software action or behavior specified in a second natural language response. In other words, the second natural language response may semantically describe what software adjustments should be made to the source code files of a scientific instrument to correct any malfunction that has interrupted the instrument workflow, and the coding script may be configured to actually implement those software adjustments.

[0039] Herein, the level of precision, completeness, or specificity or detail expressed by the first natural language response, the second natural language response, or the coding script may be increased or otherwise improved by enabling the LLM to take into account supplementary information about the scientific instrument.

[0040] As another non-limiting example, a document repository may exist containing multiple documents. In various cases, each of the multiple documents may be any suitable electronic file (e.g., a Word-Doc file, a Portable Document Format (PDF) file, or a web page file) that can describe, explain, or otherwise present in text (or, in some cases, graphically or numerically) any suitable technical information relating to the design, manufacture, operation, maintenance, or troubleshooting of any suitable scientific instrument (for example, the multiple documents may be service manuals or technical handbooks (or parts thereof) for several different scientific instruments). In various cases, any of the multiple documents may be written (e.g., via any suitable word processing software, computer-aided design software, or quantitative analysis software) by a technician or engineer tasked with the task of designing, developing, prototyping, revising, manufacturing, or investigating any suitable scientific instrument. Note that in some cases, any document may be of any suitable length or size, or otherwise (e.g., it may be one or a few pages long, dozens of pages long, or hundreds of pages long). In any case, the model component can electronically search the document repository for one or more documents substantially related to an error message or device runtime data. In some embodiments, the model component can do this through embedding search. For example, the encoder portion of the LLM can be used to generate specific embeddings for error messages and device runtime data. The encoder portion can be used to generate embeddings for each document in the document repository, and any document whose embedding is closest to or most similar to a particular embedding can be considered related to the error message or device runtime data.Therefore, in various aspects, model components can concatenate their relevant documents together with error messages and instrument runtime data, and the model components can generate a first natural language response, a second natural language response, or a coding script by running LLM on that expanded concatenation (in addition to any appropriate prompts). In various cases, the relevant documents can be considered to provide LLM with deeper or richer information about how the scientific instrument is known or expected to perform various instrument workflows, how the scientific instrument is known or expected to respond to various types of malfunctions, how the scientific instrument should be repaired when each malfunction occurs, or how snippets of the scientific instrument's source code should be written or formatted, and such deeper or richer information can enable LLM to make the first natural language response, the second natural language response, or the coding script more accurate or detailed. It should be understood or recognized that this can be considered a use of RAG.

[0041] In either case, the first natural language response, the second natural language response, or the coding script can be considered collectively as a solution to any malfunction that disrupted the equipment workflow.

[0042] In various embodiments, the solutions generated by the LLM can be considered easy or not difficult to implement (e.g., by changing only a few software parameters and making minor adjustments to hardware components). In such situations, the execution component of the computerized tool can electronically present the solution to the user of the scientific instrument in any preferred manner. As an unrestricted example, the execution component can visually render the first or second natural language response on any preferred computer screen or computer monitor associated with the scientific instrument so that the user can see or read the first or second natural language response. As another unrestricted example, the execution component can audibly reproduce the first or second natural language response on any preferred speaker associated with the scientific instrument (e.g., via any preferred text-to-speech conversion technique) so that the user can hear the first or second natural language response, or so that the user can hear the first or second natural language response. In this way, the computerized tool can be considered to communicate or instruct the user on what interrupted the instrument workflow and how the instrument workflow can be corrected. If the solution includes a coding script, the execution component can electronically execute the coding script (or electronically instruct the scientific instrument to execute it). Therefore, any software adjustments inferred by the LLM that enable the completion of the instrument workflow can be implemented. After the coding script has been executed, the workflow component may, in some cases, instruct the scientific instrument to re-execute or retry the instrument workflow.

[0043] However, in other embodiments, the solution generated by the LLM may instead be considered difficult or impractical to implement (e.g., extensive editing or modification of the scientific instrument's source code, extensive redesign of the scientific instrument's physical structure). In such situations, the execution component of the computerized tool can instead electronically present the solution to a field engineer, technical expert, or scientific researcher associated with the manufacturer of the scientific instrument in any preferred form (e.g., via visual or auditory representation). In such situations, the first and second natural language responses can be considered to eliminate the need for brainstorming for malfunction diagnosis or resolution on the part of the field engineer, technical expert, or scientific researcher. In addition, in such situations, the coding script can be considered to reduce the amount of manual coding that the field engineer, technical expert, or scientific researcher would need to perform to resolve any malfunction specified in the first natural language response. For example, instead of starting coding from scratch, the field engineer, technical expert, or scientific researcher can instead make any edits or changes that they deem appropriate for the coding script. In either case, providing field engineers, technical experts, or scientific researchers with a first natural language response, a second natural language response, and a coding script can significantly reduce the amount of time or effort that field engineers, technical experts, or scientific researchers must expend in troubleshooting scientific instruments (for example, when developing or releasing repair patches or updates for scientific instruments).

[0044] In some embodiments, as described above, one or more technical documents relating to the design, manufacture, operation, or repair of a scientific instrument can be fed to the LLM as RAG references to help the LLM more reliably or accurately synthesize a solution (e.g., a first natural language response, a second natural language response, or a coding script). In some embodiments, a field engineer, technical expert, or scientific researcher can provide feedback on the appropriateness or accuracy of the solution to a computerized tool (e.g., via any suitable human-computer interface of the scientific instrument). In various cases, the feedback may be any suitable electronic data (e.g., structured or unstructured text) that identifies or specifies one or more troubleshooting inaccuracies or missteps present in or included in the solution. In various cases, a model component can utilize such feedback to help the LLM prevent those same troubleshooting inaccuracies or missteps from occurring in the future. Specifically, an edit prompt may exist, which may be unstructured or plain text requesting or instructing how any RAG reference on which the LLM relied to generate the solution should be edited to avoid any inaccuracies or missteps specified in the feedback. In various embodiments, a model component can concatenate the solution, the RAG reference, and the edit prompt together. In various cases, a model component can feed its concatenation into the input layer of the LLM, which can complete a forward pass through one or more hidden layers of the LLM, and the output layer of the LLM can compute one or more text edits based on the activations provided by one or more hidden layers of the LLM.

[0045] In various cases, one or more text edits may be synthesized semantic markup that substantially satisfies the editing prompt. In other words, one or more text edits may be word deletions or word insertions, and if word deletions or word insertions are applied to the RAG reference before the LLM generates the solution, the solution will not have whatever troubleshooting inaccuracies or missteps indicated in the feedback. In other words, the feedback can be seen as specifying which parts of the solution were incorrectly synthesized, the LLM can be seen as predicting or inferring which pages, paragraphs, or sentences in the RAG reference caused the LLM to generate or create the incorrect parts of the solution, and the LLM can be seen as further predicting or inferring how to linguistically edit or modify those pages, paragraphs, or sentences in the RAG reference to avoid making the same mistakes, inaccuracies, or missteps in the future. In some cases, this can be seen as a reverse RAG technique (for example, this can be seen as modifying or editing a reference using feedback on synthesized text, rather than using the reference to synthesize text). In either case, the execution component can electronically implement or apply one or more text edits to whatever RAG references the LLM relies on when synthesizing the solution. Therefore, future solution synthesis performed by the LLM is more likely to be correct or more accurate (for example, one or more text edits can be seen as correcting whatever parts of the RAG reference led the LLM astray, thereby reducing the likelihood of the LLM being led astray).

[0046] In this way, computerized tools can be seen as facilitating rapid and effective automated troubleshooting in response to disruptions in scientific instrument workflows. Consequently, the amount of time that users of scientific instruments must wait for hardware or software corrections to be provided in response to malfunctions of their instruments, as well as the amount of effort that instrument manufacturers must spend brainstorming and developing such hardware or software corrections, can be significantly reduced.

[0047] Using the various embodiments described herein, hardware or software can be used to solve problems that are inherently highly technical (e.g., to facilitate large-scale language model solving of scientific instrument workflow interruptions), not abstract, and that cannot be performed as a set of mental actions by humans. Furthermore, some of the processes to be performed can be carried out by dedicated computers for performing defined actions related to the field of charged particle microscopy (e.g., electron microscopes such as SEM, TEM, or EELM), mass spectrometers coupled to liquid, gas, or ion chromatographs, or artificial neural networks (e.g., LLM) for performing defined actions related to the field of scientific instrumentation.

[0048] For example, such a defined operation may include: causing a scientific instrument (e.g., a charged particle microscope, a mass spectrometer) to perform a workflow on a sample via a device operably coupled to a processor; retrieving runtime data logged by the scientific instrument during the workflow, in response to the scientific instrument generating an error message that interrupts the workflow; and synthesizing a first text explaining why the workflow was interrupted, via the execution of a large-scale language model on the error message and runtime data, via the device. Such a defined operation may further include: synthesizing a second text explaining how to resolve the workflow, via the execution of a large-scale language model on the first text, via the device. In various embodiments, such a defined operation may further include: synthesizing a coding script configured to cause the scientific instrument to resolve the workflow, via the execution of a large-scale language model on the second text; and causing the scientific instrument to execute the coding script and thereby resolve the workflow. In various cases, the workflow may be an imaging scan of a sample, a milling operation of a sample, movement of an operable stage holding the sample, a spectroscopic scanning of a sample, or an injection or flushing operation of a sample. In various embodiments, runtime data may include a partial charged particle image or partial mass spectrum of the sample captured by the scientific instrument during the workflow, or the state of one or more configurable operating parameters of the scientific instrument during the workflow.In various cases, a large-scale language model can synthesize a first text in a search-enhanced generation manner based on performing an embedded search through a repository of technical documentation associated with the design, manufacture, operation, or troubleshooting of scientific instruments, and the defined tasks may further include the device receiving feedback from a user or technician regarding the first text, the device synthesizing text edits to documents in the repository of technical documentation retrieved by the embedded search, via the execution of the large-scale language model on the feedback, and the device updating the repository of technical documentation by inserting text edits into the documents.

[0049] Such defined operations are inherently computerized. Indeed, scientific instruments such as charged particle microscopes (e.g., SEM, TEM, EELM, dual-beam microscopes), or mass spectrometers coupled to chromatographs (e.g., liquid chromatographs, gas chromatographs, ion chromatographs), are highly specialized computerized devices equipped with specific computerized hardware (e.g., temperature sensors, pressure sensors, voltage sensors, ion beam emitters, electron beam emitters, focusing lenses, ion detectors, electron detectors, beam apertures, fluid valves, and operable sample stages). Scientific instruments, the operations they perform, and the electronic data they capture cannot be implemented in any reasonable or practical way by human mind or by human beings using pen and paper without a computer. Furthermore, artificial neural networks (e.g., LLMs) are also inherently computerized constructs equipped with specific software-oriented architectures (e.g., input layers, hidden layers, or output layers, any of which may consist of trainable or untrainable internal parameters such as convolutional layers or LSTM layers). Artificial neural networks cannot be trained or run in any reasonable or practical way without a computer, either by human mind or simply by human beings using pen and paper.

[0050] Furthermore, the various embodiments described herein can incorporate various teachings relating to the field of scientific instruments into practical applications. As described above, scientific instruments are highly complex and intricate devices that often require extensive professional training, education, or certification to learn how to operate, maintain, or troubleshoot them. Consequently, when a scientific instrument malfunctions, the user may not be able to confidently troubleshoot the malfunction and instead have to wait for troubleshooting to be carried out by a technical service engineer employed by the manufacturer of the scientific instrument. Unfortunately, it can take hours or even days before a technical service engineer begins to consider the malfunction. Moreover, once the technical service engineer finally begins to consider the malfunction, the time required for them to diagnose and resolve it can range from just minutes to weeks or even months, depending on the complexity of the malfunction. In fact, low-complexity malfunctions can be easily resolved by simply changing the values ​​of a few configurable operating parameters of a scientific instrument, which may take only a few minutes; moderate-complexity malfunctions may require more extensive editing of the scientific instrument's source code, which may take several days; and high-complexity malfunctions may require creating an entire new page of source code from scratch (or even redesigning the scientific instrument's hardware), which may take several weeks or even months.

[0051] The various embodiments described herein can help improve this problem by implementing large-scale language model (LLM) resolution for scientific instrument workflow interruptions. Specifically, the various embodiments described herein can leverage LLM to reduce the amount of time or effort spent troubleshooting scientific instrument malfunctions. As described herein, the various embodiments can achieve such goals by interruption-triggered LLM resolution synthesis or feedback-conditional LLM editing of RAG references.

[0052] In practice, when performing a workflow (e.g., imaging scan, spectroscopic scan, sample repositioning operation, sample heating or cooling operation), a scientific instrument may be abruptly or suddenly interrupted by an error message, thereby leaving the workflow incomplete. Various embodiments described herein can, in response to such interruptions, collect any runtime data generated or utilized by the scientific instrument during the partial execution of the workflow (e.g., partial or incomplete images, partial or incomplete spectra, currently assigned operating parameter values, auxiliary sensor feeds), and feed both the error message and runtime data as input to the LLM. When such input is accompanied by appropriate prompts (e.g., causal prompts, resolution prompts, coding prompts as described herein), as described herein, this can cause the LLM to synthesize text content that works to troubleshoot the error message. In some cases, the synthesized text content may explain what is thought to have caused the error message. In other cases, the synthesized text content may explain how to correct or eliminate the root cause of the error message. In other cases, the synthesized text content may include executable lines of computer code configured to correct or eliminate the root cause of the error message. In any case, such synthesized text content can be created in just a few seconds after the error message occurs. Compare this to the time, days, weeks, or months that a field engineer or technical service specialist might spend evaluating, brainstorming, and developing a solution, fix, or patch for the error message. In addition, as described herein, an LLM can generate such synthesized text content by relying on one or more RAG references containing technical information about scientific instruments. Occasionally, RAG references may become outdated.In some embodiments, various configurations may involve leveraging the LLM to update or edit RAG references in response to feedback indicating that the synthesized text content is somewhat inadequate. This concept can be viewed as a kind of reverse RAG and can help prevent the LLM from repeating troubleshooting errors identified by the feedback.

[0053] Furthermore, it must be emphasized how excellent and counterintuitive the various embodiments described herein are. In fact, the various embodiments described herein can be considered as conditioned for troubleshooting advice or LLM synthesis of solutions for runtime data collected by scientific instruments during interrupted or incomplete workflows. Such runtime data may, in some cases, include incomplete or partially captured electronic images, incomplete or partially captured chromatograms, or incomplete or partially captured mass spectra. Conventionally, such incomplete or partially captured data has been considered corrupted, meaningless, and otherwise useless. Therefore, conventional techniques discard such incomplete or partially captured data. In all contrast, the inventors of the various embodiments described herein have recognized that such incomplete or partially captured data, despite being incomplete, may contain potentially valuable clues about what interrupted the scientific instrument's workflow. In other words, the inventors have recognized that whatever particular type of malfunction occurs in a scientific instrument, it may leave hidden traces, signatures, or patterns in such incomplete or partially captured data, and that LLM can utilize such hidden traces, signatures, or patterns to generate troubleshooting reasoning about the scientific instrument (for example, different types of malfunctions, and therefore different types of malfunction resolution, may reveal or reveal incomplete or partially captured data in their respective ways). Since conventional wisdom teaches that such incomplete or partially captured data is useless and should be discarded, the various embodiments described herein that utilize such incomplete or partially captured data to reduce the amount of time or effort required for troubleshooting can be considered highly counterintuitive, unexpected, unique, or excellent.

[0054] For at least the reasons stated above, the various embodiments described herein can be considered to address or improve upon various problems or shortcomings related to troubleshooting scientific instruments. Therefore, the various embodiments described herein can be considered concrete and tangible technical improvements in the field of scientific instruments. Accordingly, the various embodiments described herein are certainly qualified as useful and practical applications of computers.

[0055] Furthermore, the various embodiments described herein can control real-world tangible devices based on the disclosed teachings. For example, the various embodiments described herein can electronically activate, deactivate, or otherwise operate real-world hardware (e.g., ion beam emitters, ion focusing lenses, carrier fluid valves / pumps) of real-world scientific instruments (e.g., SEMs, TEMs, EELMs, dual-beam microscopes).

[0056] Figure 1 illustrates an exemplary and non-limiting block diagram of the scientific instrument module 102 according to various embodiments described herein.

[0057] In various embodiments, the scientific instrument module 102 may be implemented by circuits (e.g., including electrical or optical components) such as programmed computing devices. The logic of the scientific instrument module 102 may reside in a single computing device or be distributed across multiple computing devices communicating with each other as needed. Examples of computing devices that can implement the scientific instrument module 102, individually or in combination, are discussed herein with reference to Figures 18 and 20, and examples of systems or networks of interconnected computing devices in which the scientific instrument module 102 can be implemented across one or more computing devices are discussed herein with reference to Figure 19.

[0058] The scientific instrument module 102 may include a first logic 104, a second logic 106, and a third logic 108. As used herein, the term “logic” may include devices that perform a set of operations associated with the logic. For example, any of the logic elements included in the scientific instrument module 102 may be implemented by one or more computing devices programmed with instructions that cause one or more processing devices of a computing device to perform a set of operations associated with the logic. In certain embodiments, the logic element may include one or more non-temporary computer-readable media having instructions that, when executed by one or more processing devices of the computing device, cause one or more computing devices to perform the associated set of operations. As used herein, the term “module” may refer to a collection of one or more logic elements that together perform a function associated with a module. Different logic elements within a module may take the same form or different forms. For example, some logic within a module may be implemented by programmed general-purpose processing devices, while other logic within a module may be implemented by application-specific integrated circuits (ASICs). In another example, different logic elements within a module may be associated with different sets of instructions executed by one or more processing devices. A module may omit one or more logic elements depicted in the relevant drawings, and for example, a module may include a subset of the logic elements depicted in the relevant drawings if the module performs a subset of the operations considered herein by reference to that module.

[0059] In various embodiments, there may be scientific instruments corresponding to the scientific instrument module 102. In various aspects, the scientific instrument may be any suitable computerized device capable of electronically measuring several scientifically relevant, clinically relevant, or research-related properties, characteristics, or attributes of an analytical sample (e.g., a known or unknown mixture, compound, or aggregate of substances). As an unrestricted example, the scientific instrument may be a scanning electron microscope. In such a case, the scientific instrument may measure or determine the surface topography of the analytical sample. As another unrestricted example, the scientific instrument may be a scanning electron microscope. In such a case, the scientific instrument may measure or determine the internal structure details of the analytical sample. As yet another unrestricted example, the scientific instrument may be an electron energy loss microscope. In such a case, the scientific instrument may measure or determine location-by-location counts or intensities over a defined range of energy loss bins or bands for the analytical sample. As a more general and non-limiting example, a scientific instrument could be any suitable type of charged particle microscope (for example, some types of microscopes can use a beam of non-electron ions to capture an image or energy spectrum, or otherwise interact with a sample). As another non-limiting example, a scientific instrument could be a mass spectrometer operably coupled to a chromatograph. In such a case, the scientific instrument can measure or determine a chromatogram of the analytical sample (e.g., relative compound abundance as a function of retention time) or an ion spectrum (e.g., relative ion abundance as a function of mass-to-charge ratio).

[0060] In various embodiments, the first logic 104 can cause the scientific instrument to perform a workflow (e.g., imaging scan, spectroscopic screening, milling, heating, cooling, rearrangement) on a sample.

[0061] In various embodiments, the second logic 106 may include retrieving runtime data (e.g., partially captured images, partially captured spectra, operating parameter values) logged by the charged particle microscope during the workflow in response to the scientific instrument generating an error message that interrupts the workflow.

[0062] In various embodiments, the third logic 108 may include synthesizing a first text explaining why the workflow was interrupted, via the execution of a large-scale language model on error messages and runtime data. In various embodiments, the third logic 108 may further include synthesizing a second text explaining how to resolve the workflow, via the execution of a large-scale language model on the first text. In various cases, the third logic 108 may further include synthesizing a coding script configured by the device, and via the execution of a large-scale language model on the second text, to cause the scientific instrument to resolve the workflow. In some cases, the large-scale language model may synthesize the first text, the second text, or the coding script in a search-enhanced generative manner, based on performing an embedded search through a repository of technical documentation associated with the design, manufacture, operation, or troubleshooting of the scientific instrument. In such a situation, the third logic 108 may further include receiving feedback from a user or engineer regarding the first text, the second text, or the coding script; synthesizing text edits to documents in the repository of technical documents retrieved by embedded search via the execution of a large language model on the feedback; and updating the repository of technical documents by inserting the text edits into the documents.

[0063] Therefore, the scientific instrument module 102 can facilitate the resolution of large-scale language models of scientific instrument workflow interruptions.

[0064] Figure 2 is an illustrative and non-limiting flowchart of the computer implementation method 200 according to various embodiments described herein. The operation of the computer implementation method 200 can be used in any preferred context to perform any preferred operation (for example, by or in conjunction with any of the various modules, computing devices, or graphical user interfaces described with respect to Figures 1, 18, 19, and 20). Although the operations are illustrated once each in a specific order in Figure 2, the operations may be rearranged or repeated as desired and as necessary (for example, different operations to be performed may be performed in parallel as appropriate).

[0065] In various embodiments, operation 202 may include performing a first operation by a device operably coupled to the processor, causing a scientific instrument (e.g., a charged particle microscope) to perform a workflow on a sample. In various cases, the first logic 104 may perform or otherwise facilitate operation 202.

[0066] In various embodiments, operation 204 may include performing a second operation, which, in response to the device and the scientific instrument generating an error message that interrupts the workflow, retrieves runtime data logged by the scientific instrument during the workflow. In various cases, the second logic 106 may perform operation 204 or otherwise facilitate it.

[0067] In various cases, action 206 may include the device performing a third action, which synthesizes a first text explaining why the workflow was interrupted, through the execution of a large language model on error messages and status data.

[0068] Therefore, the computer implementation method 200 can facilitate the resolution of large-scale language models of scientific instrument workflow interruptions.

[0069] Figure 3 illustrates a block diagram of an exemplary, non-limiting system that facilitates large-scale language model resolution of scientific instrument workflow interruptions, according to one or more embodiments described herein.

[0070] In various embodiments, the scientific instrument 302 may exist. In various aspects, the scientific instrument 302 may be as described above. That is, the scientific instrument 302 may be any suitable computerized device that can electronically capture any suitable type of measurable or quantifiable data of any suitable analytical sample by utilizing its constituent hardware (e.g., electron source, anode, condenser lens, condenser aperture, scanning coil, objective lens, objective aperture, deflector, capacitor, stigmatall, electron detector, X-ray detector, operable sample stage, autosump injector, oven heating column, mass spectrometer, absorbent packing material, stationary phase film, vacuum chamber, fluid conduit). As a non-limiting example, the scientific instrument 302 may be any suitable type of charged particle microscope (e.g., SEM, TEM, EELM, dual-beam microscope) that can electronically capture or generate any suitable type of charged particle image of the surface or internal portion of the analytical sample. As another non-limiting example, the scientific instrument 302 may be any suitable chromatograph (e.g., liquid chromatograph, gas chromatograph, ion chromatograph, thin-layer chromatograph, affinity chromatograph) or mass spectrometer (e.g., quadrupole mass spectrometer, time-of-flight mass spectrometer, ion trap mass spectrometer, Fourier transform ion cyclotron resonance mass spectrometer, magnetic sector mass spectrometer) capable of electronically capturing any suitable type of compositional ion abundance data (e.g., chromatogram, mass spectrum) related to the analyte.

[0071] Although not explicitly shown in the diagram, the scientific instrument 302 may be electronically integrated with any suitable human-computer interface device, which may be remote or local to the scientific instrument 302. Thus, a user or technician associated with the scientific instrument 302 may interact with or otherwise control the scientific instrument 302. Some non-limiting examples of human-computer interface devices may be a keyboard for the scientific instrument 302, a keypad for the scientific instrument 302, a touchscreen for the scientific instrument 302, or a voice command system for the scientific instrument 302.

[0072] In any case, the scientific instrument 302 may include a plurality of configurable operating parameters 304. In various embodiments, each of the plurality of configurable operating parameters 304 may be any preferred hardware-related or software-related characteristic of the scientific instrument 302 that guides, influences, or otherwise determines how the scientific instrument 302 runs, operates, or functions with respect to any given analytical sample, and can be selectively controlled, modified, adjusted, or otherwise set by a user or technician (e.g., through interaction with the human-computer interface device of the scientific instrument 302). As a non-limiting example, any of the plurality of configurable operating parameters 304 may be a user-controllable voltage setting (e.g., beam voltage) or current setting (e.g., beam current) that allows a user or technician to selectively control the electrodes of the scientific instrument 302 so as to selectively increase or decrease a voltage or current within or applied by the scientific instrument 302. As another non-limiting example, one of the multiple configurable operating parameters 304 could be a user-configurable temperature setting, which allows a user or technician to control a heater (e.g., a stage heater, heating coil) or cooler (e.g., a cooling fan, heat pump, refrigerator) in the scientific instrument 302 to selectively increase or decrease the temperature within or applied by the scientific instrument 302. As yet another non-limiting example, one of the multiple configurable operating parameters 304 could be a user-configurable mechanical actuator setting, which allows a user or technician to control a mechanical actuator (e.g., an electric motor, sample stage, aperture, fluid pump, or syringe) in the scientific instrument 302 to selectively move the mechanical actuator.As yet another non-limiting example, one of several configurable operating parameters 304 may be a user-controllable optical setting that allows a user or technician to control the optical elements of the scientific instrument 302 (e.g., optical lenses, optical deflectors) to selectively change the optical quality applied by the scientific instrument 302 (e.g., focal size or position, astigmatism, defocus).

[0073] In various embodiments, the scientific instrument 302 may have a currently loaded sample 306, or may be otherwise associated with it. In various cases, the currently loaded sample 306 may be (as its name suggests) currently loaded on or inside the scientific instrument 302. In an unrestricted example, in a situation where the scientific instrument 302 is a charged particle microscope, the currently loaded sample 306 may be currently positioned, located, or otherwise attached to the operable stage of the scientific instrument 302, such that the currently loaded sample 306 can be analyzed or scanned by the scientific instrument 302. In another unrestricted example, in a situation where the scientific instrument 302 is a chromatograph or mass spectrometer, the currently loaded sample 306 may be currently positioned, located, or otherwise housed within the autosampler injector of the scientific instrument 302, such that the currently loaded sample 306 can be analyzed or scanned by the scientific instrument 302. In various cases, the currently loaded sample 306 may be any suitable type of medical, clinical, scientific, or laboratory sample that can exhibit any suitable physical, chemical, compositional, or other properties, attributes, or characteristics. As an unrestricted example, the currently loaded sample 306 may be a lamellar taken from a semiconductor substrate or wafer. As another unrestricted example, the currently loaded sample 306 may be a sample or fragment taken from a defective load-bearing structure. As yet another unrestricted example, the currently loaded sample 306 may be an aqueous solution or mixture containing any suitable type of solvent or solute.

[0074] In various embodiments, a large-scale language model 308 (hereinafter referred to as "LLM308") may exist. In various embodiments, the LLM308 may comprise an encoder portion 310 and a synthesizer portion 312. In various cases, the encoder portion 310 may be considered upstream of the synthesizer portion 312. Similarly, the synthesizer portion 312 may be considered downstream of the encoder portion 310.

[0075] In various embodiments, the encoder portion 310 can exhibit any suitable deep learning internal architecture. In fact, in various cases, the encoder portion 310 may have an input layer, one or more hidden layers, and an output layer. In various cases, any of these layers may be connected together by any suitable interneuronal or interlayer connections, such as forward connections, skip connections, or recursive connections. Furthermore, in various cases, any of these layers may be any suitable type of neural network layer having any suitable learnable or trainable internal parameters. For example, any of the input layer, one or more hidden layers, or output layer may be a convolutional layer whose learnable or trainable parameters may be a convolutional kernel. As another example, any of the input layer, one or more hidden layers, or output layer may be a dense layer whose learnable or trainable parameters may be a weight matrix or bias values. As yet another example, any of the input layer, one or more hidden layers, or output layer may be a batch normalization layer whose learnable or trainable parameters may be shift coefficients or scale coefficients. As yet another example, any of the input layer, one or more hidden layers, or output layer may be an LSTM layer whose learnable or trainable parameters are an input state weight matrix or a hidden state weight matrix. As yet another example, any of the input layer, one or more hidden layers, or output layer may be a transformer layer whose learnable or trainable parameters are a single-head or multi-head attention block or other weight matrix. Furthermore, in various cases, any of the layers may be any suitable type of neural network layer having any suitable fixed or untrainable intrinsic parameters. For example, any of the input layer, one or more hidden layers, or output layer may be a nonlinear layer, a padding layer, a pooling layer, or a concatenated layer.

[0076] Similarly, in various cases, the synthesizer portion 312 can exhibit any suitable deep learning internal architecture. In fact, in various cases, the synthesizer portion 312 may have an input layer, one or more hidden layers, and an output layer. In various cases, any of these layers may be connected together by any suitable interneuronal or interlayer connections (e.g., forward connections, skip connections, or recursive connections). Furthermore, in various cases, any of these layers may be any suitable type of neural network layer having any suitable learnable or trainable intrinsic parameters (e.g., any of the input layer, one or more hidden layers, or output layer may be a convolutional layer, a high-density layer, a batch normalization layer, an LSTM layer, or a transformer layer). Moreover, in various cases, any of these layers may be any suitable type of neural network layer having any suitable fixed or untrainable intrinsic parameters (e.g., any of the input layer, one or more hidden layers, or output layer may be a nonlinear layer, a padding layer, a pooling layer, or a concatenated layer).

[0077] Regardless of the specific internal architecture implemented within encoder section 310 (e.g., specific number, type, or organization of layers), encoder section 310 may be configured to receive text data (which may be accompanied by any preferred numerical or graphical data) and to generate embeddings based on such input text data. In contrast, regardless of the specific internal architecture implemented within synthesizer section 312, synthesizer section 312 may be configured to receive embeddings created by encoder section 310 and to create synthesized text content based on such embeddings. As some non-limiting examples, LLM308 may be any of the following: ChatGPT, Generative AI, Ollama, Bard, or Claude.

[0078] In various embodiments, system 314 can be electronically integrated with the scientific instrument 302 and LLM 308 (for example, via any preferred wired or wireless electronic connection). As described herein, system 314 can facilitate troubleshooting of the scientific instrument 302 by leveraging LLM 308 (for example, reducing the number of days, weeks, or months spent troubleshooting or solution development to just seconds or minutes).

[0079] In various embodiments, the system 314 may include a processor 316 (e.g., a computer processing unit, a microprocessor) and a non-temporary computer-readable memory 318 that is operable, operable, or communicatively connected to or coupled to the processor 316. The non-temporary computer-readable memory 318 can store computer-executable instructions that, when executed by the processor 316, cause the processor 316 or other components of the system 314 (e.g., a workflow component 320, a status component 322, a model component 324, an execution component 326) to perform one or more operations. In various embodiments, the non-temporary computer-readable memory 318 can store computer-executable components (e.g., a workflow component 320, a status component 322, a model component 324, an execution component 326), and the processor 316 can execute the computer-executable components.

[0080] In various embodiments, system 314 can electronically access the scientific instrument 302 or LLM 308. That is, the access component 314 can electronically communicate with or otherwise electronically interact with the scientific instrument 302 or LLM 308 (e.g., send electronic instructions or commands to it, receive electronic data from it). Thus, any suitable component of system 314 can interact with, communicate with, or otherwise operate the scientific instrument 302 or LLM 308. It should be noted that system 314 may, in some cases, be implemented on or hosted on the scientific instrument 302 itself or on any suitable computerized workstation associated with or coupled to the scientific instrument 302. In such situations, system 314 can be considered deployed in a client-side manner (e.g., system 314 may be considered local to the scientific instrument 302). However, in other cases, system 314 may instead be implemented or hosted remotely from the scientific instrument 302, such as in a cloud computing environment. In such a situation, system 314 can be considered to be deployed in the server-side format.

[0081] In various embodiments, the system 314 may include a workflow component 320. In various aspects, the workflow component 320 may cause the scientific instrument 302 to initiate an instrument workflow, as described herein.

[0082] In various embodiments, system 314 may include a status component 322. In various cases, the status component 322 may collect runtime data generated or utilized by the scientific instrument 302 in response to the instrument workflow being interrupted by an error message, as described herein.

[0083] In various embodiments, system 314 may include model component 324. In various cases, model component 324 can troubleshoot error messages that interrupt the equipment workflow by leveraging runtime data and LLM 308, as described herein.

[0084] In various embodiments, the system 314 may include an execution component 326. In various aspects, the execution component 326 may render, share, or otherwise implement any troubleshooting results generated by the LLM, as described herein.

[0085] It should be noted that in various cases, the workflow component 320, the status component 322, the model component 324, and the execution component 326 may be considered collectively as one or more software components 319 of the system 314. It should be understood that in various embodiments, for the sake of ease of explanation and illustration, one or more software components 319 are described herein primarily as comprising four components (e.g., the workflow component 320, the status component 322, the model component 324, and the execution component 326). However, one or more software components 319 are not limited to being implemented strictly as such four components in all embodiments. In fact, in some embodiments, the functions described herein for such four components may be combined in any preferred manner so as to be implemented in fewer than four components (for example, in some cases, a single component may perform all of the functions described herein with respect to the workflow component 320, the status component 322, the model component 324, and the execution component 326). In other embodiments, the functions of such four components described herein may instead be distributed, separated, divided, or fragmented in any preferred manner so as to be implemented in or by more than four components (for example, two or more components may facilitate a function that can be performed by the workflow component 320, two or more components may facilitate a function that can be performed by the status component 322, two or more components may facilitate a function that can be performed by the model component 324, and two or more components may facilitate a function that can be performed by the execution component 326).

[0086] Figure 4 illustrates an exemplary, non-limiting system block diagram of an instrument workflow, including error messages, that can facilitate large-scale language model resolution of scientific instrument workflow interruptions, according to one or more embodiments described herein.

[0087] In various embodiments, the workflow component 320 can electronically command, instruct, or otherwise electronically trigger the scientific instrument 302 to start, initiate, or otherwise initiate the execution of the instrument workflow 402. In various embodiments, the instrument workflow 402 may be any preferred sequence of one or more hardware or software operations that are implementable, conductable, or performable by the scientific instrument 302. As a non-limiting example, in a situation where the scientific instrument 302 is a charged particle microscope, the instrument workflow 402 may be a charged particle imaging protocol. In such a case, the instrument workflow 402 can be considered to specify any sequence of operations or actions that would produce a desired charged particle image (e.g., a pixel array, a voxel array) of the surface or internal volume of the currently loaded sample 306 (e.g., establishing a desired focal size, establishing a desired beam voltage or current, establishing a desired beam raster pattern). As another non-limiting example, in a situation where the scientific instrument 302 is a charged particle microscope, the instrument workflow 402 may be a milling protocol. In such cases, the instrument workflow 402 can be considered to specify any sequence of operations or actions (e.g., reducing the focal size, increasing the beam voltage or current, directing the beam to a desired position on the currently loaded sample 306, or injecting a reactive gas onto the currently loaded sample 306) to remove or etch a desired amount of material from the currently loaded sample 306 (or deposit a desired amount of material onto the sample 306). In yet another non-limiting example, in situations where the scientific instrument 302 is a charged particle microscope, the instrument workflow 402 may be a spatial repositioning protocol. In such cases, the instrument workflow 402 can be considered to specify any sequence of operations or actions (e.g., establishing a desired angular or translational position for the operable stage of the scientific instrument 302, or extending, retracting, or otherwise articulating the robotic arm of the scientific instrument 302 in a desired manner) to move or reorient the currently loaded sample 306 in any desired manner.As yet another non-limiting example, in a situation where the scientific instrument 302 is a charged particle microscope, the instrument workflow 402 may be a vacuum protocol. In such a case, the instrument workflow 402 can be considered to specify any sequence of operations or actions that cause the vacuum chamber of the scientific instrument 302 to achieve any desired pressure (e.g., opening and closing the load lock door of the scientific instrument 302, operating or stopping the vacuum pump of the scientific instrument 302). As yet another non-limiting example, in a situation where the scientific instrument 302 is a chromatograph or mass spectrometer, the instrument workflow 402 may be a scanning or screening protocol. In such a case, the instrument workflow 402 can be considered to specify any sequence of operations or actions that cause the scientific instrument 302 to scan or screen the sample 306 currently loaded for any desired chemical compound or molecule (e.g., heating the column oven of the scientific instrument 302 to a desired temperature, activating the injection pump or syringe of the scientific instrument 302, establishing a desired ion monitoring range for the scientific instrument 302). As another non-limiting example, in situations where the scientific instrument 302 is a chromatograph or mass spectrometer, the instrument workflow 402 could be a sample heating protocol. In such a case, the instrument workflow 402 can be considered to specify any sequence of actions or movements (e.g., heating the autosampler of the scientific instrument 302) that causes the scientific instrument 302 to bring the currently loaded sample 306 to a desired temperature. As yet another non-limiting example, in situations where the scientific instrument 302 is a chromatograph or mass spectrometer, the instrument workflow 402 could be a sample flushing protocol.In such cases, the sample 306 currently loaded may be a cleaning agent rather than a research curiosity, and therefore the instrument workflow 402 can be considered to specify any sequence of actions or behaviors that cause the scientific instrument 302 to flush or clean its internal conduits or pipes in a desired manner using the sample 306 currently loaded (e.g., establishing a desired fluid flow rate or fluid pressure in the autosampler injector of the scientific instrument 302).

[0088] Therefore, the instrument workflow 402 may be any preferred set of automated hardware or software actions that can be performed by the scientific instrument 302 to achieve some desired result or effect with respect to the currently loaded sample 306. However, these are merely non-limiting examples. In some embodiments, the currently loaded sample 306 may be omitted. That is, the scientific instrument 302 may be empty or otherwise have no sample loaded inside it, but it may nevertheless be desired for the scientific instrument 302 to perform the instrument workflow 402 (for example, there may be some workflows that are desired or intended to be performed without a sample).

[0089] In various cases, the instrument workflow 402 may be selected or specified by the user or technician associated with the scientific instrument 302. As a non-limiting example, the user or technician may interact with or otherwise use the human-computer interface of the scientific instrument 302 to select an instrument workflow 402 from a defined list of available instrument workflows. As another non-limiting example, the user or technician may interact with or otherwise use the human-computer interface of the scientific instrument 302 to define an instrument workflow 402 in a customized or ad-hoc manner.

[0090] In either case, the workflow component 320 can cause the scientific instrument 302 to initiate the execution of any sequence of actions that constitutes the instrument workflow 402. In various embodiments, the scientific instrument 302 may be abruptly, suddenly, unexpectedly, or otherwise undesirably interrupted by an error message 404 while executing the instrument workflow 402. Non-limiting details will be described with respect to FIG. 5.

[0091] FIG. 5 illustrates an exemplary non-limiting block diagram showing how an error message 404 can interrupt an instrument workflow 402 according to one or more embodiments described herein.

[0092] Assume that the instrument workflow 402 is a sequence of x operations for any suitable positive integer x > 1, where each of such x operations can be executed or performed by the scientific instrument 302. In response to a command or instruction from the workflow component 320, the scientific instrument 302 can begin to progress in that manner through the sequence of its x operations, starting from the first operation in the sequence and ending with the xth operation in the sequence. During the execution of the instrument workflow 402, a malfunction may occur in the scientific instrument 302. Such a malfunction can cause the scientific instrument 302 to pause the instrument workflow 402 and instead electronically generate, create, or display an error message 404. For example, assume that a malfunction occurs when the scientific instrument 302 attempts to perform the jth operation of the instrument workflow 402 for any suitable positive integer j < x. In response to the malfunction, the scientific instrument 302 can refrain from proceeding to the (j + 1)th operation of the instrument workflow 402 and instead create or throw an error message 404. In other words, the malfunction can be considered to cause the scientific instrument 302 to pause the instrument workflow 402 such that the instrument workflow 402 remains unfinished or incomplete.

[0093] In various aspects, error message 404 may be any suitable electronic data exhibiting any suitable format, size, or dimension that can be considered to convey, indicate, or otherwise represent any suitable information, characteristics, features, or attributes about or relating to any malfunction that interrupted the instrument workflow 402 (e.g., it may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, or any suitable combination thereof). In a non-limiting example, error message 404 may be any suitable structured text or alphanumeric identifier that indicates that any component of the hardware or software of the scientific instrument 302 has ceased to function properly, without specifying what is wrong with that hardware or software component (e.g., a first alphanumeric identifier may uniquely indicate that the oven of the scientific instrument 302 has ceased to function properly, but may not specify what is wrong with the oven, and a second alphanumeric identifier may uniquely indicate that the ion beam emitter of the scientific instrument 302 has ceased to function properly, but may not specify what is wrong with the ion beam emitter). As another non-limiting example, error message 404 could be any suitable structured text or alphanumeric identifier indicating that a malfunction of a particular type, category, or class has occurred in each component of the hardware or software of the scientific instrument 302 (for example, a first alphanumeric identifier could uniquely indicate that the oven of the scientific instrument 302 has overheated, and a second alphanumeric identifier could uniquely indicate that the oven of the scientific instrument 302 has been physically damaged). As yet another non-limiting example, error message 404 could be any suitable unstructured text describing that a malfunction of a particular type, category, or class has occurred in each component of the hardware or software of the scientific instrument 302 (for example, one or more natural language sentences or sentence fragments that semantically state or describe any suitable information regarding which part of the scientific instrument 302 has malfunctioned and in what manner).More generally, different components of the scientific instrument 302 may fail or malfunction in different ways, and the error message 404 may represent or communicate such information to any preferred level of specificity or generality (for example, the error message 404 may indicate that the scientific instrument 302 has malfunctioned, the error message 404 may indicate that a particular part of the scientific instrument 302 has malfunctioned, or the error message 404 may indicate that a particular type of malfunction has occurred in a particular part of the scientific instrument 302).

[0094] In either case, the workflow component 320 can cause the scientific instrument 302 to start executing the instrument workflow 402, and the error message 404 (and therefore whatever malfunction caused the error message 404) can be considered to interrupt the instrument workflow 402.

[0095] Figure 6 illustrates an exemplary, non-limiting block diagram of a system containing instrument runtime data that can facilitate large-scale language model resolution of scientific instrument workflow interruptions, according to one or more embodiments described herein.

[0096] In various embodiments, the status component 322 may electronically instruct, command, or otherwise electronically trigger the scientific instrument 302 to extract, record, or recall instrument runtime data 602 in response to an interruption of the instrument workflow 402 by an error message 404. In various cases, the instrument runtime data 602 may be any electronic data logged or otherwise utilized by the scientific instrument 302 during or in connection with its partial execution of the instrument workflow 402. Various non-limiting embodiments will be described with reference to Figure 7.

[0097] Figure 7 illustrates an exemplary and non-limiting block diagram showing device runtime data 602 according to one or more embodiments described herein.

[0098] In various embodiments, the instrument runtime data 602 may include a plurality of configurable operating parameter states 702. In various embodiments, the plurality of configurable operating parameter states 702 may correspond (for example, in a one-to-one manner) to a plurality of configurable operating parameters 304. More specifically, each of the plurality of configurable operating parameter states 702 may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, or any preferred combination thereof that constitutes or represents any state or value that one of the plurality of configurable operating parameters 304 had or was assigned during a partial execution of the instrument workflow 402. As an unrestricted example, the plurality of configurable operating parameter states 702 may indicate whatever focal size the scientific instrument 302 used when attempting the instrument workflow 402. As another unrestricted example, the plurality of configurable operating parameter states 702 may indicate whatever beam voltage the scientific instrument 302 used when attempting the instrument workflow 402. As yet another non-limiting example, multiple configurable operating parameter states 702 could indicate whatever oven temperature the scientific instrument 302 was using when it was attempting the instrument workflow 402.

[0099] In various embodiments where the scientific instrument 302 is a charged particle microscope and the instrument workflow 402 is an imaging protocol, the instrument runtime data 602 may include a partially captured charged particle image 704. In various embodiments, the partially captured charged particle image 704 may be any preferred partially filled or partially generated pixel array or voxel array created by the scientific instrument 302 during its partial implementation of the instrument workflow 402. For example, it may be desired that the instrument workflow 402 yields an SEM image illustrating or illustrating the sample 306 currently being loaded. However, due to being interrupted by an error message 404, the instrument workflow 402 may instead yield an incomplete SEM image with a majority of pixels still having their default values ​​(e.g., still being zero when they should not), and that incomplete SEM image may be considered a partially captured charged particle image 704. In other words, the scientific instrument 302 may be in the process of rasterizing the currently loaded sample 306 when the instrument workflow 402 is interrupted, and therefore the partially captured charged particle image 704 may have a large blank swath representing any region of the currently loaded sample 306 that was expected or intended to be rasterized but has not yet been rasterized. Conventional techniques must take note that the partially captured charged particle image 704 is considered wasteful or corrupted and therefore useless.

[0100] In various embodiments where the scientific instrument 302 is a chromatograph or mass spectrometer and the instrument workflow 402 is a scanning or screening protocol, the instrument runtime data 602 may include partially captured spectral data 706. In various embodiments, the partially captured spectral data 706 may be any preferred partially filled or partially generated chromatogram or mass spectrum created by the scientific instrument 302 during its partial implementation of the instrument workflow 402. For example, it may be desired that the instrument workflow 402 yield a peak-by-peak chromatogram (e.g., abundance versus retention time) and the respective mass spectrum (e.g., abundance versus mass-to-charge ratio) for the sample 306 currently loaded, with a maximum retention time of 120 seconds implemented. However, due to interruption by error message 404, the instrument workflow 402 may instead only collect chromatographic and mass spectral data for the first 60 seconds of retention time. Therefore, any chromatograms and mass spectra obtained by the instrument workflow 402 will be missing any chromatographic or ion peaks that occur only when the retention time exceeds 60 seconds. Such incomplete or unfinished chromatograms or mass spectra can be collectively considered as partially captured spectral data 706. Conventional techniques must emphasize that partially captured spectral data 706 is wasteful or corrupted and therefore useless.

[0101] In various embodiments, the instrument runtime data 602 may include one or more in-instrument pressure feeds 708. In particular, the scientific instrument 302 may, in various embodiments, be equipped with or otherwise fitted with one or more integrated or built-in pressure sensors. Non-limiting examples of such integrated or built-in pressure sensors may include strain gauges, capacitive pressure transducers, resistive pressure transducers, piezoelectric pressure transducers, or optical pressure transducers. In any case, each of such integrated or built-in pressure sensors may continuously, sequentially, or periodically monitor what solid or fluid pressure each hardware component of the scientific instrument 302 experiences or encounters. Thus, each of the one or more in-instrument pressure feeds 708 may be any time series of pressure measurements recorded by each integrated or built-in pressure sensor during the partial execution of the instrument workflow 402.

[0102] In various embodiments, the instrument runtime data 602 may include one or more in-instrument video feeds 710. In particular, the scientific instrument 302 may, in various embodiments, be equipped with one or more integrated or built-in video cameras, or otherwise. Non-limiting examples of such integrated or built-in video cameras may include endoscopes, thermal imaging cameras, night vision cameras, or any suitable type of visible spectrum camera. In any case, each of such integrated or built-in video cameras may continuously, sequentially, or periodically monitor the visual appearance of each hardware component of the scientific instrument 302. Thus, each of the one or more in-instrument video feeds 710 may be any time series of video frames recorded by each integrated or built-in video camera during the partial execution of the instrument workflow 402.

[0103] In various embodiments, the instrument runtime data 602 may include one or more in-instrument audio feeds 712. In particular, the scientific instrument 302 may, in various embodiments, be equipped with one or more integrated or built-in microphones, or otherwise fitted with them. Non-limiting examples of such integrated or built-in microphones may include dynamic microphones, condenser microphones, ribbon microphones, piezoelectric microphones, or electret microphones. In any case, each of such integrated or built-in microphones may continuously, sequentially, or periodically monitor noise emitted by each hardware component of the scientific instrument 302. Thus, each of the one or more in-instrument audio feeds 712 may be any time series of audio data recorded by each integrated or built-in microphone during the partial execution of the instrument workflow 402.

[0104] In various embodiments, the instrument runtime data 602 may include one or more in-instrument temperature feeds 714. In particular, the scientific instrument 302 may, in various embodiments, be equipped with one or more integrated or built-in temperature sensors, or otherwise. Non-limiting examples of such integrated or built-in video temperature sensors may include thermocouples, thermistors, semiconductor-based thermometers, thermal infrared sensors, or thermal diodes. In any case, each of such integrated or built-in temperature sensors may continuously, sequentially, or periodically monitor the temperature occurring in each hardware component of the scientific instrument 302. Thus, each of the one or more in-instrument temperature feeds 714 may be any time series of temperature measurements recorded by each integrated or built-in temperature sensor during the partial execution of the instrument workflow 402.

[0105] In various embodiments, the instrument runtime data 602 may include one or more in-instrument humidity feeds 716. In particular, the scientific instrument 302 may, in various embodiments, be equipped with one or more integrated or built-in humidity sensors, or otherwise. Non-limiting examples of such integrated or built-in humidity sensors may include capacitive hygroscopes, resistive hygroscopes, optical hygroscopes, gravimetric hygroscopes, piezoelectric hygroscopes, or solid-state hygroscopes. In any case, each of such integrated or built-in humidity sensors can continuously, sequentially, or periodically monitor the humidity occurring in each hardware component of the scientific instrument 302. Thus, each of the one or more in-instrument humidity feeds 716 may be a time series of humidity measurements recorded by each integrated or built-in humidity sensor during the partial execution of the instrument workflow 402.

[0106] Figure 7 should be understood or recognized as merely an unrestricted example of instrument runtime data 602. In some cases, any of the following may be excluded or omitted from the instrument runtime data 602 based on data availability or as desired: multiple configurable operating parameter states 702, partially captured charged particle images 704, partially captured spectral data 706, one or more in-instrument pressure feeds 708, one or more in-instrument video feeds 710, one or more in-instrument audio feeds 712, one or more in-instrument temperature feeds 714, or one or more in-instrument humidity feeds 716.

[0107] In either case, the status component 322 can electronically collect or retrieve instrument runtime data 602 from the scientific instrument 302 in response to the instrument workflow 402 being interrupted by the error message 404.

[0108] Figure 8 illustrates an exemplary, non-limiting system block diagram including a technical documentation repository and solutions that can facilitate large-scale language model solutions for scientific instrument workflow interruptions, according to one or more embodiments described herein.

[0109] In various embodiments, the model component 324 may electronically store, maintain, control, or otherwise access (e.g., locally or remotely) the technical documentation repository 802. In various embodiments, the model component 324 may electronically generate a solution 804 to the error message 404 by utilizing the LLM 308, the device runtime data 602, or the technical documentation repository 802. Various non-limiting details are described with reference to Figures 9 to 14.

[0110] Figures 9 to 14 illustrate exemplary, non-limiting block diagrams showing how a solution 804 to an error message 404 can be generated based on the device runtime data 602 and the technical documentation repository 802, according to one or more embodiments described herein.

[0111] First, consider Figure 9. In various cases, a workflow indicator 902 may exist. In various cases, the workflow indicator 902 may be any suitable electronic data exhibiting any suitable format, size, or dimension that can be considered to convey, indicate, or otherwise represent any suitable information, characteristics, or attributes related to the equipment workflow 402 (for example, it may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, or any suitable combination thereof). As a non-limiting example, the workflow indicator 902 may indicate identifying information of the equipment workflow 402, or any suitable structured text or alphanumeric identifier that uniquely corresponds thereto. In some cases, the workflow indicator 902 and the error message 404 can be considered collectively to indicate the total number of steps, actions, or operations involved in the equipment workflow 402, and at which of those steps, actions, or operations the equipment workflow 402 was interrupted.

[0112] In various embodiments, the model component 324 can electronically execute the LLM 308 in response to device runtime data 602, error messages 404, workflow indicators 902, or any preferred combination thereof. In various cases, such execution can cause the LLM 308 to create some synthesized text 906.

[0113] More specifically, the model component 324 can concatenate the device runtime data 602, error messages 404, and workflow indicators 902 together. Note that in some cases, such concatenation may include any other combinations described above (for example, fewer than all of the device runtime data 602, error messages 404, and workflow indicators 902). In various cases, the model component 324 can feed or route its concatenation to the input layer of the encoder section 310. In various cases, its concatenation may complete a forward path through one or more hidden layers of the encoder section 310. In various embodiments, the output layer of the encoder section 310 may compute the embedding 904 based on the activation map or feature map provided by one or more hidden layers of the encoder section 310, or compute it otherwise.

[0114] In various cases, the embedding 904 can be considered a latent vector representation that the encoder portion 310 considers or infers to correspond to the concatenation of the instrument runtime data 602, the error message 404, and the workflow indicator 902. More specifically, the embedding 904 may be one or more scalars, one or more vectors, one or more matrices, one or more tensors, or any preferred combination thereof. In various embodiments, the number of dimensions of the embedding 904 (e.g., the total number or density of numerical elements in the embedding 904) may be smaller (e.g., by several orders of magnitude) than the total or cumulative number of dimensions of the instrument runtime data 602, the error message 404, and the workflow indicator 902. In various cases, even if the embedding 904 has a smaller number of dimensions, the embedding 904 can still be considered to represent at least some substantial or semantic content of the instrument runtime data 602, the error message 404, and the workflow indicator 902, albeit in a hidden or invisible manner. In other words, the embedded 904 can be considered a compact or compressed numerical representation of the device runtime data 602, error message 404, and workflow indicator 902. It should be noted that a third party without connection to or relationship with the encoder portion 310 cannot reproduce or infer the device runtime data 602, error message 404, and workflow indicator 902 from the embedded 904 alone, and therefore the embedded 904 can be considered to represent the device runtime data 602, error message 404, and workflow indicator 902 in a latent, ambiguous, or otherwise hidden form.

[0115] In various embodiments, the embedding 904 may be fed or routed to the input layer of the synthesizer section 312. In various embodiments, the embedding 904 may complete a forward path through one or more hidden layers of the synthesizer section 312. In various embodiments, the output layer of the synthesizer section 312 may compute or otherwise compute the embedding 906 based on the activation map or feature map provided by one or more hidden layers of the synthesis section 312.

[0116] In various embodiments, the synthesized text 906 may be one or more declarative sentences or sentence fragments generated by the synthesis portion 312 based on the embedding 904. It should be noted that the synthesized text 906 does not necessarily have to be an estimation or approximate reconstruction of the instrument runtime data 602, the error message 404, and the workflow indicator 902. Instead, the synthesized text 906 may be any number of suitable synthesized sentences that are semantically or substantially related in some way to the embedding 904, and therefore to the instrument runtime data 602, the error message 404, and the workflow indicator 902. In some cases, the synthesized text 906 can be considered to include hallucinations that are semantically or substantially related to the instrument runtime data 602, the error message 404, and the workflow indicator 902.

[0117] In various embodiments, the model component 324 may ignore, discard, or delete the synthesized text 906. However, the model component 324 may record, save, store, or otherwise maintain the embedding 904. In other words, the model component 324 may extract the embedding 904 from the encoder portion 310 (for example, from the hidden layer of LLM 308).

[0118] Now, let us consider Figure 10. In various embodiments, the technical documentation repository 802 may contain multiple technical documents 1002. In various embodiments, the multiple technical documents 1002 may contain n documents for any suitable positive integer n>1, i.e., documents 1002(1) to 1002(n). In various embodiments, each of the multiple technical documents 1002 may be any suitable electronic file (e.g., a Word-Doc file, a PDF file, a web page file) that describes, teaches, shows, indicates, or otherwise communicates, in text (or in some cases graphically or numerically), one or more technical features, details, or aspects of any suitable scientific instrument (which may or may not include scientific instrument 302), or one or more technical features, details, or aspects of any suitable instrument workflow (which may or may not include instrument workflow 402).As some non-limiting examples, any of several technical documents 1002 may describe the technical or scientific design of any suitable charged particle microscope, chromatograph, or mass spectrometer (e.g., different components, parts, or subsystems of various instruments may be listed or shown, and how those components, parts, or subsystems operate); how any suitable charged particle microscope operates (e.g., different user-configurable settings or controls of various microscopes may be listed or shown, and what such settings or controls do); any suitable information regarding the expected use or operation of any suitable charged particle microscope, chromatograph, or mass spectrometer (e.g., various operating conditions or use case scenarios that the instruments are designed to handle or not handle may be shown or described); and how any suitable charged particle microscope, chromatograph, or mass spectrometer is expected or assumed to be maintained. This may be a service manual, maintenance handbook, schematic diagram, failure mode report, or any part thereof, describing or explaining any suitable information (e.g., listing, displaying, or describing various repair tasks that may be expected to be performed on various instruments), any suitable information regarding troubleshooting of any suitable charged particle microscope, chromatograph, or mass spectrometer (e.g., describing or explaining how to resolve various malfunction symptoms of various instruments), or any suitable information regarding workflows commonly performed by charged particle microscopes, chromatographs, or mass spectrometers (e.g., listing, describing, or explaining how different types of instrument workflows should be performed, listing, describing, or explaining the requirements or prerequisites for different instrument workflows, listing, describing, or explaining how different instrument workflows commonly fail, and listing, describing, or explaining how to correct common problems in different instrument workflows).Some other non-limiting examples include any of the technical documents 1002, any source code script or file relating to any suitable charged particle microscope, chromatograph, or mass spectrometer, any suitable maintenance ticket or repair ticket relating to any suitable charged particle microscope, chromatograph, or mass spectrometer, or any suitable customer complaint form or customer comment form relating to any suitable charged particle microscope, chromatograph, or mass spectrometer.

[0119] It should be recognized and understood that any of the technical documents 1002 may be any sub-part or subsection of some larger document. For example, any of the technical documents 1002 may be a chapter, section, paragraph, or even a sentence from some longer document.

[0120] In various embodiments, the model component 324 can electronically generate multiple embeddings 1004 by performing the LLM 308 as described above for each of the multiple technical documents 1002.

[0121] As a non-limiting example, model component 324 can run LLM308 on technical document 1002(1), and model component 324 can extract embedding 1004(1) from LLM308 during its execution. More specifically, model component 324 can feed or route technical document 1002(1) to the input layer of encoder portion 310, technical document 1002(1) can complete a forward pass through one or more hidden layers of encoder portion 310, and the output layer of encoder portion 310 can compute or otherwise compute embedding 1004(1) based on the activation map or feature map provided by one or more hidden layers of encoder portion 310. It should be noted that embedding 1004(1) can have the same format, size, or number of dimensions as embedding 904 (for example, embedding can be a vector of uniform dimensions or size representing sentences, paragraph embeddings can be aggregated or averaged together to produce paragraph embeddings, paragraph embeddings can be aggregated or averaged together to produce section or chapter embeddings, and section or chapter embeddings for the entire document can be aggregated or averaged together to produce embeddings for the entire technical document), and therefore embedding 1004(1) can be considered a latent vector representation of the technical document 1002(1). In various cases, embedding 1004(1) can then complete a forward path through synthesizer part 312, but model component 324 can ignore, discard, or delete any synthesized text content that synthesizer part 312 creates based on embedding 1004(1).

[0122] As another non-restrictive example, model component 324 can run LLM308 on technical document 1002(n), and model component 324 can extract embedding 1004(n) from LLM308 during its execution. In fact, as described above, model component 324 can feed or route technical document 1002(n) to the input layer of encoder portion 310, technical document 1002(n) can complete a forward pass through one or more hidden layers of encoder portion 310, and the output layer of encoder portion 310 can compute or otherwise compute embedding 1004(n) based on the activation map or feature map provided by one or more hidden layers of encoder portion 310. Thus, embedding 1004(n) can have the same format, size, or number of dimensions as embedding 904, and therefore embedding 1004(n) can be considered a latent vector representation of technical document 1002(n). As described above, the embedding 1004(n) can then complete a forward pass through the synthesizer section 312, but the model component 324 may ignore, discard, or delete any synthesized text content that the synthesizer section 312 creates based on the embedding 1004(n).

[0123] In various cases, embeddings 1004(1) to 1004(n) can be collectively considered as multiple embeddings 1004.

[0124] In various embodiments, the model component 324 can electronically determine or identify a set of relevant technical documents 1006 by comparing the embedding 904 with a plurality of embeddings 1004. In particular, for each given embedding of the plurality of embeddings 1004, the model component 324 can calculate any suitable error or similarity value between the given embedding and embedding 904. In some non-limiting examples, such error or similarity values ​​may be calculated using mean absolute error (MAE), mean squared error (mean The calculation may involve squared error (MSE), cosine similarity, Euclidean distance, or cross-entropy. In any case, the model component 324 can conclude that a set of related technical documents 1006 is either the one whose embedding (e.g., in 906) is most similar to embedding 904, or one of the closest set of technical documents 1002. As a non-limiting example, the model component 324 can identify any m of the set of technical documents 1002 that are most similar to embedding 904, or have the closest embedding, and such m documents can be considered a set of related technical documents 1006 for any suitable positive integer m. That is, a set of related technical documents 1006 can include m documents, i.e., related technical documents 1006(1) to related technical documents 1006(m). In other words, related technical document 1006(1) can be any of several technical documents 1002 whose embedding is the closest to or most similar to embedding 904, whereas related technical document 1006(m) can be any of several technical documents 1002 whose embedding is the m-th closest to or most similar to embedding 904.

[0125] In any case, the set of relevant technical documents 1006 can be considered to be substantially or semantically related in some way to the instrument runtime data 602, the error message 404, or the workflow indicator 902. As a non-limiting example, suppose the workflow indicator 902 specifies that the instrument workflow 402 is a particular imaging protocol, and the error message 404 indicates that the ion beam emitter of the scientific instrument 302 has stopped functioning properly. In such cases, any of the set of relevant technical documents 1006 may be structured or unstructured text describing or explaining what components or hardware or software constitute the scientific instrument 302, what sequence of imaging operations constitutes the instrument workflow 402, how each of those imaging operations is performed, executed, or implemented, what focal size, raster pattern, or beam voltage the instrument workflow 402 requires, how the ion beam emitter is designed, functions, or constructed, what types of malfunctions generally affect the ion beam emitter, what types of malfunctions generally result in partially captured images, partially captured spectral data, in-instrument pressure measurements, in-instrument video recordings, in-instrument audio recordings, in-instrument temperature measurements, or in-instrument humidity measurements similar to those shown in the instrument runtime data 602, or how to troubleshoot or correct any of such malfunctions. Thus, the set of relevant technical documents 1006 can be considered to provide valuable contextual information regarding the instrument runtime data 602, error messages 404, or workflow indicators 902.

[0126] Next, we consider Figure 11. In various embodiments, a causal prompt 1102 may exist. In various embodiments, the causal prompt 1102 may be one or more unstructured text or plain text sentences or sentence fragments requesting or commanding that any malfunction that interrupts the instrument workflow 402 be identified, determined, or diagnosed for a specific hardware or software cause. In fact, the scientific instrument 302 may consist of numerous hardware or software components (e.g., an ion beam emitter, an electron detector, a collimating lens, a vacuum pump, an operating system, a downloaded library), and any given part of the hardware or software may be subject to numerous types of failures or malfunctions (e.g., an ion beam emitter may be subject to cathode failure, anode failure, or insulator failure), and a given type of failure or malfunction may have different underlying causes or root causes (e.g., an insulator failure in an ion beam emitter may be caused by dielectric breakdown via a voltage surge, dielectric breakdown due to overheating; dielectric breakdown due to vibration fatigue; dielectric breakdown due to moisture absorption; or dielectric breakdown due to gas release). Therefore, the causal prompt 1102 can be seen as requesting identification of the specific, granular, underlying, or root cause of whatever malfunction interrupted the device workflow 402. As a non-limiting example, the causal prompt 1102 could be the statement: "The workflow specified by the workflow indicator 902. The workflow interrupted by error message 404, resulting in device runtime data 602. What specific malfunction interrupted the workflow?" Another non-limiting example would be the statement: "The workflow specified by the workflow indicator 902. The workflow interrupted by error message 404, resulting in device runtime data 602. Please diagnose the cause of the interruption."

[0127] Therefore, in various cases, the model component 324 can electronically execute the LLM 308 on the device runtime data 602, on the error message 404, on the workflow indicator 902, on the related technical document 1006, on the cause prompt 1102, or on any preferred combination thereof. In various cases, such execution can cause the LLM 308 to generate the explanatory text 1104. More specifically, the model component 324 can concatenate the device runtime data 602, the error message 404, the workflow indicator 902, the set of related technical documents 1006, and the causal relationship prompt 1102 together. Note that, as described above, in various cases, the concatenation can include any other combination mentioned above (for example, fewer than all of the device runtime data 602, the error message 404, the workflow indicator 902, the set of related technical documents 1006, and the causal relationship prompt 1102). In various cases, the model component 324 can feed its concatenation to the input layer of the encoder section 310. In various embodiments, the concatenation can complete a forward path through one or more hidden layers of the encoder section 310. In various cases, the output layer of the encoder section 310 can compute or otherwise calculate one or more embeddings (not shown) based on the activation maps or feature maps provided by one or more hidden layers of the encoder section 310. In various cases, one or more of these embeddings can be routed to the input layer of the synthesizer section 312. In various embodiments, one or more of these embeddings can complete a forward path through one or more hidden layers of the synthesizer section 312, and the output layer of the synthesizer section 312 can compute or otherwise calculate a set of descriptive texts 1104 based on the activation maps or feature maps provided by one or more hidden layers of the synthesizer section 312.

[0128] In various embodiments, the explanatory text 1104 may be one or more declarative or sentence fragments of unstructured or plain text that semantically respond to the causal prompt 1102. That is, the explanatory text 1104 may be a composite text that explains or states what underlying problem with the scientific instrument 302 is thought to have caused the instrument workflow 402 to be interrupted. In other words, the explanatory text 1104 may be natural language or plain text that describes or explains what particular hardware-related or software-related problem caused the scientific instrument 302 to generate the error message 404 midway through the instrument workflow 402. In various cases, the workflow indicator 902 may be considered to indicate known information about the instrument workflow 402, while the error message 404 and instrument runtime data 602 (enhanced or supplemented by the set of related technical documents 1006) may be considered measured or detected information about any unknown malfunction or failure that interrupted the instrument workflow 402. In some cases, the unknown malfunction or failure may leave a trace, signature, or other unique effect in the instrument runtime data 602. Ultimately, different underlying failures may manifest differently (for example, some failures may manifest visually, which may uniquely affect a partially captured charged particle image 704 or one or more in-instrument video feeds 710; others may manifest aurally, which may uniquely affect one or more in-instrument audio feeds 712; and yet another may manifest thermally, which may uniquely affect one or more in-instrument temperature feeds 714). Thus, LLM 308 can be considered to recognize what trace, signature, or unique effect the unknown malfunction has left in the instrument runtime data 602, and such recognition can therefore enable the inference of identifying information of the unknown malfunction. Accordingly, the descriptive text 1104 can semantically convey or indicate that identifying information.

[0129] Now, consider Figure 12. In various embodiments, a resolution prompt 1202 may exist. In various embodiments, the resolution prompt 1202 may be one or more unstructured text or plain text sentences or sentence fragments requesting or commanding that a corrective action be identified or determined for whatever particular malfunction is specified in the explanatory text 1104. Indeed, as stated above, a given type of failure or malfunction may have different root causes, and different root causes may require different corrective actions (for example, dielectric breakdown due to a voltage surge may be repaired or addressed by a first sequence of troubleshooting or repair actions; in contrast, dielectric breakdown due to overheating may not respond to a first sequence of troubleshooting or repair actions and may instead be repaired or addressed via a second sequence of troubleshooting or repair actions). Thus, the resolution prompt 1202 can be considered, when executed on the scientific instrument 302, as requesting the identification of a specific action or protocol to resolve or correct whatever has interrupted the instrument workflow 402. As a non-restrictive example, resolution prompt 1202 could be the following sentence: "The workflow specified by workflow indicator 902. The workflow interrupted by error message 404, resulting in device runtime data 602. The inferred cause of the interruption indicated by descriptive text 1104. How do you resolve the inferred cause?" Another non-restrictive example would be the following sentence: "The workflow specified by workflow indicator 902. The workflow interrupted by error message 404, resulting in device runtime data 602. The inferred cause of the interruption indicated by descriptive text 1104. Decide how to correct it."

[0130] Therefore, in various cases, the model component 324 can electronically execute the LLM 308 on the equipment runtime data 602, on the error message 404, on the workflow indicator 902, on the set of related technical documents 1006, on the explanatory text 1104, on the resolution prompt 1202, or on any preferred combination thereof. In various cases, such execution can cause the LLM 308 to create the resolution text 1204. More specifically, the model component 324 can concatenate the equipment runtime data 602, the error message 404, the workflow indicator 902, the set of related technical documents 1006, the explanatory text 1104, and the resolution prompt 1202 together. Note that, as described above, in various cases, the concatenation can include any other combination of the aforementioned (for example, fewer than all of the equipment runtime data 602, the error message 404, the workflow indicator 902, the set of related technical documents 1006, the explanatory text 1104, or the resolution prompt 1202). In various cases, the model component 324 can feed its concatenation to the input layer of the encoder section 310. In various embodiments, its concatenation can complete a forward path through one or more hidden layers of the encoder section 310. In various embodiments, the output layer of the encoder section 310 can compute or otherwise calculate one or more embeddings (not shown) based on the activation maps or feature maps provided by one or more hidden layers of the encoder section 310. In various cases, one or more of these embeddings can be routed to the input layer of the synthesizer section 312. In various embodiments, one or more of these embeddings can complete a forward path through one or more hidden layers of the synthesizer section 312, and the output layer of the synthesizer section 312 can compute or otherwise calculate the resolved text 1204 based on the activation maps or feature maps provided by one or more hidden layers of the synthesizer section 312.

[0131] In various cases, the resolution text 1204 may be one or more declarative or sentence fragments of unstructured or plain text that semantically respond to the resolution prompt 1202. That is, the resolution text 1204 may be a composite text that explains or describes what repair action or sequence of operations will resolve or correct any underlying problem relating to the scientific instrument 302 specified in the explanatory text 1104. In other words, the resolution text 1204 may be natural language or plain text that forms a tutorial instructing what particular hardware or software modification, when applied to the scientific instrument 302, will resolve or correct a specific hardware or software-related problem that interrupted the instrument workflow 402. Non-limiting examples of hardware repair modifications include replacing consumable fluids, cartridges, or component parts of the scientific instrument 302; tightening or loosening specific bolts or fasteners of the scientific instrument 302; drying, wetting, or lubricating specified surfaces or lenses of the scientific instrument 302; leveling, reorienting, or repositioning the scientific instrument 302; or adding structural support to the scientific instrument 302. Non-limiting examples of software repair modifications include rebooting or restarting the scientific instrument 302; adjusting one of several configurable operating parameters 304 to a specific value; or editing the source code files of the scientific instrument 302. In any of these cases, LLM308 can be considered to determine how to resolve whatever caused the interruption of the equipment workflow 402 to the error message 404, by utilizing the inferences of LLM308 itself as shown in the explanatory text 1104, and any traces, signatures, or unique patterns present in the equipment runtime data 602 (supplemented or enhanced by the set of related technical documents 1006). Thus, the resolution text 1204 can semantically communicate or indicate that determination.

[0132] Next, consider Figure 13. In various embodiments, a code prompt 1302 may exist. In various aspects, the code prompt 1302 may be one or more unstructured or plain text sentences or sentence fragments that require or instruct a coding script to be generated or written to implement any software modifications specified in the resolution text 1204. As a non-limiting example, the code prompt 1302 may be the sentence: "The workflow specified by the workflow indicator 902. The workflow interrupted by the error message 404, resulting in the device runtime data 602. The inferred cause of the interruption indicated by the description text 1104. The resolution indicated in the resolution text 1204. Write a script to implement the resolution."

[0133] Therefore, in various cases, the model component 324 can electronically execute the LLM 308 on the device runtime data 602, on the error message 404, on the workflow indicator 902, on the set of related documents 1006, on the explanatory text 1104, on the resolution text 1204, on the code prompt 1302, or on any preferred combination thereof. In various cases, such execution can cause the LLM 308 to create the synthesized code 1304. More specifically, the model component 324 can concatenate the device runtime data 602, the error message 404, the workflow indicator 902, the set of related technical documents 1006, the explanatory text 1104, the resolution text 1204, and the code prompt 1302 together. As described above, it should be noted that in various cases the concatenation may include any other combination of the aforementioned (for example, it may include fewer than all of the instrument runtime data 602, error messages 404, workflow indicators 902, the set of related technical documents 1006, explanatory text 1104, resolution text 1204, or code prompts 1302). In various cases, the model component 324 may feed its concatenation to the input layer of the encoder section 310. In various embodiments, its concatenation may complete a forward pass through one or more hidden layers of the encoder section 310. In various cases, the output layer of the encoder section 310 may compute or otherwise compute one or more embeddings (not shown) based on the activation maps or feature maps provided by one or more hidden layers of the encoder section 310. In various cases, one or more of these embeddings may be routed to the input layer of the synthesizer section 312. In various embodiments, one or more of these embeddings can complete a forward pass through one or more hidden layers of the synthesizer portion 312, and the output layer of the synthesizer portion 312 can compute or otherwise compute the synthesized code 1304 based on the activation map or feature map provided by one or more hidden layers of the synthesizer portion 312.

[0134] In various embodiments, the synthesized code 1304 may be one or more structured texts that semantically address or satisfy the resolution prompt 1202. That is, the synthesized code 1304 may be a coding or programming script configured to perform or implement at runtime any repair software modification specified in the resolution text 1204. In other words, the synthesized code 1304 may be one or more lines of computer executable code (e.g., consisting of variable definitions, function calls, for loops, if loops, or while loops) that, when written in any preferred programming syntax and executed by a computing device, cause the LLM 308 to perform any software-related action that resolves anything that interrupts the equipment workflow 402.

[0135] As shown in Figure 14, solution 804 can be considered to include or consist of explanatory text 1104, resolution text 1204, and synthesized code 1304.

[0136] It should be noted that in some embodiments, the solution 804 may omit or exclude any of the explanatory text 1104, the solution text 1204, or the synthesized code 1304.

[0137] Figures 9 to 14 illustrate how LLM 308 sequentially generates explanatory text 1104, resolution text 1204, and synthesized code 1304, but this is merely a non-limiting example for the sake of explanation and illustration. In various embodiments, there may be a single integrated prompt containing any combination of the contents of causal prompt 1102, resolution prompt 1202, or code prompt 1302. In such a case, LLM 308 may be executed on a concatenation of device runtime data 602, error messages 404, workflow indicators 902, a set of relevant technical documents 1006, and its integrated prompt, and such execution may cause LLM 308 to generate explanatory text 1104, resolution text 1204, and synthesized code 1304 simultaneously or substantially simultaneously.

[0138] Although not explicitly shown in Figures 9 to 14, it should be noted that in some embodiments in which the descriptive text 1104, the resolution text 1204, and the synthesized code 1304 are generated sequentially, the set of relevant technical documents 1006 can be updated iteratively or incrementally before each execution of the LLM 308. As a non-limiting example, the set of relevant technical documents 1006 may initially be extracted by performing an embedding search through the technical document repository 802 based on the embedding 904, as shown in the figure. However, after the descriptive text 1104 has been generated, a new embedding can be created to collectively represent the equipment runtime data 602, the error message 404, the workflow indicator 902, and the descriptive text 1104, and the technical document repository 802 can be searched again using that new embedding. Thus, whatever technical documents are retrieved from such a new search can be considered semantically relevant not only to the equipment runtime data 602, the error message 404, and the workflow indicator 902, but also to the descriptive text 1104. These new related technical documents can thus be fed as input to LLM308 to improve the accuracy or reliability of the resolution text 1204. Similarly, after the resolution text 1204 is generated, another new embedding can be created to collectively represent the equipment runtime data 602, error messages 404, workflow indicators 902, explanatory text 1104, and resolution text 1204, and the technical document repository 802 can be searched using this other new embedding. Thus, whatever technical documents are retrieved from such another new search can be considered semantically relevant not only to the equipment runtime data 602, error messages 404, workflow indicators 902, and explanatory text 1104, but also to the resolution text 1204. Therefore, these other new related technical documents can thus be fed as input to LLM308 to improve the accuracy or reliability of the synthesized code 1304.

[0139] In any case, the model component 324 can generate the solution 804 by utilizing the LLM 308, the device runtime data 602, or the technical documentation repository 802.

[0140] In various embodiments, the execution component 326 can perform any preferred subsequent actions in response to the generation of the solution 804. As an unrestricted example, the execution component 326 can electronically transmit the solution 804 (or any preferred portion thereof) to any preferred computing device. As another unrestricted example, the execution component 326 can electronically render the solution 804 (or any preferred portion thereof) on any preferred electronic computer screen or display.

[0141] It should be recognized that the difficulty of implementing solution 804 can range from easy to complex. After all, some types of malfunctions may be easy to address (e.g., changing a few operating parameter values, cleaning a specific surface of the scientific instrument 302). In contrast, other types of malfunctions may be very difficult to address (e.g., extensive editing of the source code of the scientific instrument 302, extensive modification or redesign of the physical structure of the scientific instrument 302). Therefore, in situations where solution 804 is easy to implement or not complex to implement, it may be acceptable or appropriate for the execution component 326 to electronically transmit solution 804 to a computing device associated with the end user of the scientific instrument 302, or to electronically render solution 804 on the computing device. On the other hand, in situations where the solution 804 is performed intensively or implementation is not easy, it may be acceptable or appropriate for the execution component 326 to electronically transmit the solution 804 to a computing device associated with an engineer or technical expert of the manufacturer of the scientific instrument 302, or to render the solution 804 on the computing device.

[0142] In either case, solution 804 can be generated within a few seconds or minutes after the equipment workflow 402 has been interrupted. Therefore, solution 804 can be considered to significantly accelerate the process of troubleshooting such interruptions compared to the time, days, weeks, or months required by existing techniques.

[0143] In some embodiments in which solution 804 includes synthesized code 1304, the execution component 326 can electronically instruct, command, or otherwise electronically trigger the scientific instrument 302 to execute synthesized code 1304. Thus, the scientific instrument 302 can execute any repair software fix as specified in solution text 1204. In response to such execution, the execution component 326 can, in some cases, electronically instruct, command, or electronically trigger the scientific instrument 302 to restart or retry the instrument workflow 402 to verify whether the root problem specified in description text 1104 or the root problem has been successfully repaired by the execution of synthesized code 1304. If the scientific instrument 302 experiences the same or another interruption during such restart or retry of the instrument workflow 402, the system 314 can electronically contact a technical expert by repeating the various actions described herein or by using an automated request for troubleshooting assistance.

[0144] In some embodiments, the LLM 308 can infer or predict that whatever malfunction caused the error message 404 is one that has never been seen or occurred before. In such cases, the explanatory text 1104 may indicate that the root cause of the error message 404 is unknown or has never been seen before. In such situations, the execution component 326 may, in response to the explanatory text 1104 indicating that the root cause is unknown or has never been seen before, electronically send a request for assistance to any suitable computing device of a technical expert associated with the manufacturer of the scientific instrument 302. In some such cases, the model component 324 may refrain from generating the resolution text 1204 and the synthesized code 1304. In other cases of such circumstances, the model component 324 may generate the resolution text 1204 and the synthesized code 1304, which together may be considered a potential or proposed solution to a previously unseen cause of the error message 404, and the execution component 326 may accordingly send the resolution text 1204 and the synthesized code 1304 to the computing device of a technical expert, thereby assisting the technical expert even though the root cause of the error message 404 is one that has not been previously seen.

[0145] Figure 15 illustrates an exemplary and non-limiting system block diagram including a solution feedback and a set of technical documentation editors, which can facilitate large-scale language model resolution of scientific instrument workflow interruptions, according to one or more embodiments described herein.

[0146] In various embodiments, the system 314 may electronically receive, retrieve, or otherwise access the solution feedback 1502 from any preferred source. In various embodiments, the solution feedback 1502 may be any preferred electronic data (e.g., one or more scalars, one or more vectors, one or more matrices, one or more tensors, or one or more strings) that represents or otherwise conveys the shortcomings or inadequacies of the solution 804.

[0147] As a non-limiting example, the execution component 326 may, as described above, share the solution 804 with a computing device associated with a technical expert of the manufacturer of the scientific instrument 302, or render the solution 804 on the computing device. Thus, the technical expert may read or review the solution 804. In some cases, the technical expert may identify one or more inaccurate or suboptimal parts of the solution 804 (for example, they may determine that the explanatory text 1104 describes an incorrect root cause, that the solution text 1204 describes an incorrect repair action or an incorrect order of repair actions, or that the synthesized code 1304 describes an incorrect function call). Thus, the solution feedback 1502 may be one or more declarative natural language sentences or sentence fragments written by the technical expert (for example, using any suitable word processing software) that describe or explain how or why one or more particular parts of the solution 804 are inaccurate.

[0148] In various embodiments, the model component 324 can utilize the LLM 308 and solution feedback 1502 to generate one or more technical document edits 1504. Various non-limiting embodiments are described with reference to Figure 16.

[0149] Figure 16 illustrates an exemplary and non-limiting block diagram showing how one or more technical document edits 1504 can be generated based on the solution feedback 1502 according to one or more embodiments described herein.

[0150] In various embodiments, an editing prompt 1602 may exist. In various embodiments, the editing prompt 1602 may be one or more unstructured or plain text sentences or sentence fragments that request or command the editing of each of the related technical documents in the set of related technical documents 1006 in order to avoid any inaccuracies in the reasoning specified in the resolution feedback 1502. In other words, LLM 308 may consider that the inaccuracies in the reasoning specified in the resolution feedback 1502 are caused by certain parts of the set of related technical documents 1006 (e.g., certain chapters, certain pages, certain paragraphs, certain sentences) being misled (e.g., possibly due to those documents being outdated), and the editing prompt 1602 may be considered to request LLM 308 to identify those particular parts of the set of related technical documents 1006 and to determine how those particular parts should be edited or modified so that LLM 308 is not misled in the same way again in the future. As a non-restrictive example, edit prompt 1602 could be the following sentence: "The workflow specified by workflow indicator 902. The workflow interrupted by error message 404, resulting in equipment runtime data 602. Troubleshooting inferred from solution 804 and based on a set of relevant technical documents 1006. Inaccuracies in the solution, as indicated by solution feedback 1502. What edits to the set of relevant technical documents 1006 would have avoided the inaccuracies in the solution?" As a non-restrictive example, edit prompt 1602 could be the following sentence: "The workflow specified by workflow indicator 902. The workflow interrupted by error message 404, resulting in equipment runtime data 602. Troubleshooting inferred from solution 804 and based on a set of relevant technical documents 1006. What edits would have made the set of relevant technical documents 1006 match solution feedback 1502?"

[0151] Therefore, in various cases, the model component 324 can electronically execute LLM 308 on the device runtime data 602, on the error message 404, on the workflow indicator 902, on the set of related technical documents 1006, on the solution 804, on the solution feedback 1502, on the edit prompt 1602, or on any preferred combination thereof. In various cases, such execution can cause LLM 308 to create one or more technical document edits 1504. More specifically, the model component 324 can concatenate the device runtime data 602, the error message 404, the workflow indicator 902, the set of related technical documents 1006, the solution 804, the solution feedback 1502, and the edit prompt 1602 together. As described above, it should be noted that in various cases, the concatenation may include any other combination of the aforementioned combinations (for example, it may include fewer than all of the following: instrument runtime data 602, error messages 404, workflow indicators 902, the set of related technical documents 1006, solutions 804, solution feedback 1502, or editing prompts 1602). In various cases, the model component 324 may feed its concatenation to the input layer of the encoder section 310. In various embodiments, its concatenation may complete a forward pass through one or more hidden layers of the encoder section 310. In various cases, the output layer of the encoder section 310 may compute or otherwise compute one or more embeddings (not shown) based on the activation maps or feature maps provided by one or more hidden layers of the encoder section 310. In various cases, one or more of these embeddings may be routed to the input layer of the synthesizer section 312. In various embodiments, one or more of these embeddings can complete a forward pass through one or more hidden layers of the synthesizer portion 312, and the output layer of the synthesizer portion 312 can compute one or more technical document edits 1504 based on the activation maps or feature maps provided by one or more hidden layers of the synthesizer portion 312, or compute them otherwise.

[0152] In various embodiments, one or more technical document edits 1504 may be one or more fragments of structured or unstructured text that semantically respond to or address an editing prompt 1602. That is, one or more technical document edits 1504 may describe or state one or more text edits or text modifications (e.g., deletion or insertion of a word, deletion, insertion or rephrasing of a sentence, deletion, insertion or rephrasing of a paragraph), or otherwise be a composite text that is one or more text edits or text modifications, and if these text edits or text modifications were made for each technical document in the set of related technical documents 1006, the solution 804 would have matched the solution feedback 1502. In other words, each of the one or more technical document edits 1504 could be a grammatical or semantic change, markup, or redline on each of the related technical documents 1006, and if such change, markup, or redline was made on the related technical documents before the generation of the solution 804, it prevented any inaccuracies specified in the solution feedback 1502 from being included in the solution 804. In other words, LLM 308 can be considered to infer which particular parts of which particular technical documents relied on to generate the solution 804 led LLM 308 astray or produced erroneous inferences specified in the solution feedback 1502, and LLM 308 can further infer how those particular parts of those particular technical documents should be rewritten to prevent LLM 308 from being led astray again in the future. Therefore, one or more technical document edits 1504 can be considered to be rewritten versions of those particular parts of those particular technical documents (or, otherwise, rewrite instructions for those particular parts).

[0153] In various embodiments, the execution component 326 can electronically implement one or more technical document edits 1504. In other words, the execution component 326 can electronically apply one or more technical document edits 1504 to each related technical document in the set of related technical documents 1006. In other words, the execution component 326 can semantically rewrite or modify one or more of the set of related technical documents 1006 in any manner specified by one or more technical document edits 1504. Thus, the substantial information contained in the technical document repository 802 can be considered updated, and therefore, inaccuracies in troubleshooting solutions inferred by LLM 308 in the future are more likely to be resolved.

[0154] In order to increase the likelihood that solution 804 will be accurate or correct according to the various embodiments described herein, LLM308 may be first trained. A non-limiting example of such training is described with respect to Figure 17.

[0155] Figure 17 illustrates an exemplary, non-limiting block diagram showing how the LLM308 can be trained according to one or more embodiments described herein.

[0156] In various embodiments, before training begins, the trainable internal parameters of LLM308 (e.g., convolution kernel, weight matrix, bias values) can be initialized by system 314 in any preferred manner (e.g., via random initialization).

[0157] In various embodiments, there may be training inputs 1702 and ground truth annotations 1704. In various embodiments, training input 1702 may be any suitable text data, numerical data, or graphical data that can be received by LLM308 as described herein. In some merely non-limiting examples, training input 1702 may be instrument runtime data, error messages, workflow indicators, technical documentation, prompts, any other suitable structured or unstructured text or numerical data, or any suitable combination or concatenation thereof. In various cases, ground truth annotations 1704 may be any correct or accurate synthesized text content (e.g., 1104, 1204, or 1504) or code (e.g., 1304) that is known or considered to correspond to training input 1702.

[0158] In any case, the system 314 causes the LLM 308 to run on the training input 1702, thereby causing the LLM 308 to produce the output 1706. More specifically, in some cases the training input 1702 may be fed or routed to the input layer of the LLM 308, the training input 1702 may complete a forward pass through one or more hidden layers of the LLM 308, and the output layer of the LLM 308 may compute the output 1706 based on the activation map or feature map provided by one or more hidden layers of the LLM 308.

[0159] It should be noted that the format, size, or dimensionality of output 1706 may be determined by the number, arrangement, size, or other characteristics of the neurons, convolutional kernels, attention blocks, or other internal parameters of the output layer of LLM308 (or any other layer). Therefore, output 1706 can be forced to have any desired format, size, or dimensionality by adding, removing, or otherwise adjusting the characteristics of the output layer of LLM308 (or any other layer).

[0160] In various aspects, output 1706 can be considered as predicted or inferred text content (e.g., predicted or inferred explanatory text, predicted or inferred solution text, predicted or inferred synthesized code, predicted or inferred technical documentation edit) synthesized by LLM308 based on the training input 1702. In contrast, ground truth annotation 1704 can be considered as any correct or accurate text content (e.g., correct or accurate explanatory text, correct or accurate solution text, correct or accurate synthesized code, correct or accurate technical documentation edit) that is known to or is considered to correspond to the training input 1702. Note that if LLM308 has received little or no training to date, output 1706 may be highly inaccurate. In other words, output 1706 may differ significantly from ground truth annotation 1704.

[0161] In various embodiments, the loss 1708 between the output 1706 and the ground truth annotation 1704 (e.g., MAE, MSE, cross-entropy error) can be calculated by system 314. In various cases, the trainable internal parameters of LLM308 can be progressively updated based on the loss 1708 via backpropagation (e.g., stochastic gradient descent).

[0162] In various cases, such execution and update procedures can be repeated for any suitable number of input-annotation pairs. This can ultimately lead to the iterative optimization of the LLM308's trainable internal parameters to accurately perform text synthesis or code synthesis. In various aspects, any suitable training batch size, any suitable error / loss function, or any suitable training termination criterion can be used during such training.

[0163] The disclosure herein primarily describes LLM308 as being trained in a supervised manner, but this is merely a non-limiting example for the sake of clarity and illustration. In various embodiments, LLM308 can be trained using any other suitable training paradigm, such as unsupervised training, semi-supervised training, or reinforcement learning, and any of the training paradigms may be associative or unassociative.

[0164] The disclosure herein primarily describes LLM 308 as being trained or configured to synthesize solution 804, but these are merely non-limiting examples for the sake of illustrating and illustrating. In various embodiments, LLM 308 may be configured to synthesize any suitable visual graphics that can complement solution 804. Non-limiting examples of such visual graphics may include formatted reports or presentation slides that are based on, derived from, or detail solution 804, or plots, graphs, or charts that are based on, derived from, or detail solution 804. Thus, the execution component 326 can visually render such visual graphics in addition to, or otherwise in conjunction with, solution 804.

[0165] The disclosure herein primarily describes the execution component 326 as a device that visually or audibly renders or presents the solution 804 (or any associated visual graphics) to a user or technician of the scientific instrument 302, but these are merely non-limiting examples to facilitate explanation and illustration. In various embodiments, the execution component 326 may electronically transmit the solution 804 (or any associated visual graphics) to any suitable computing device associated with the scientific instrument 302. As a non-limiting example, the execution component 326 may share the solution 804 (or any associated visual graphics) for the scientific instrument 302 or with any suitable downstream software tool or application operating in conjunction with the scientific instrument 302 (for example, some embodiments may involve transmitting the solution 804 to such downstream software tool or application rather than presenting the solution 804 to a user or technician).

[0166] In various cases, machine learning algorithms or models may be implemented in any preferred manner to facilitate any preferred embodiment described herein. To facilitate some of the above-described embodiments of machine learning in various embodiments, consider the following considerations of artificial intelligence (AI). Various embodiments described herein can use artificial intelligence to facilitate the automation of one or more features or functions. Components can use various AI-based schemes to perform the various embodiments / examples disclosed herein. To provide or assist in providing or providing many of the decisions described herein (e.g., deciding, confirming, inferring, calculating, predicting, forecasting, estimating, deriving, predicting, detecting, computing), components described herein can examine all or a subset of data to which it is permitted to access, and can provide or determine an inference about the state of a system or environment from a set of observations such as those captured through events or data. Decisions may be used, for example, to identify a particular context or action, or to generate a probability distribution over states. Decisions may be probabilistic, i.e., a calculation of a probability distribution over a state in question based on consideration of data and events. A decision can also refer to a technique used to construct a higher-level event from a set of events or data.

[0167] Such decisions can result in the construction of new events or actions from observed events or stored event data sets, regardless of whether the events are temporally close and correlated, and whether the events and data originate from one or more event and data sources. The components disclosed herein can utilize various classification schemes or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, etc.) in relation to performing automated or determined actions in connection with the claimed subject matter. Thus, several functions, actions, or decisions can be automatically learned and performed using classification schemes or systems.

[0168] The classifier uses the input attribute vector z = (z1, z2, z3, z4, z n The input can be mapped to a confidence level to which it belongs to a class, such as f(z) = confidence level (class). Such classification can use probability or statistics-based analysis (e.g., taking into account analytical utility and cost) to determine the action that should be taken automatically. A support vector machine (SVM) can be an example of a classifier that can be used. An SVM works by finding a hypersurface in space of possible inputs, which attempts to separate trigger criteria from non-trigger events. Intuitively, this correctly classifies test data that is similar to but not identical to the training data. Other directed and undirected model classification techniques include, for example, naive Bayes, Bayesian networks, decision trees, neural networks, fuzzy logic models, or probabilistic classification models that provide independence of different patterns, any of which can be used. The classifications used herein also include statistical regressions that are used to develop priority models.

[0169] To provide additional context to the various embodiments described herein, Figure 18 and the following discussion are intended to provide a brief and general description of a preferred computing environment 1800 that can implement various embodiments of the embodiments described herein. Although the embodiments are described above in the general context of computer executable instructions that can be performed on one or more computers, those skilled in the art will recognize that the embodiments can also be implemented in combination with other program modules or as a combination of hardware and software.

[0170] In general, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, those skilled in the art will understand that the methods of the present invention can be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each of which can be coupled to operate on one or more associated devices.

[0171] The illustrated embodiments of the embodiments herein can also be implemented in a distributed computing environment in which specific tasks are performed by remote processing devices linked over a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0172] Computing devices typically include a variety of media, which may include computer-readable storage media, machine-readable storage media, or communication media, and these two terms are used herein to distinguish them from one another as follows: Computer-readable storage media or machine-readable storage media can be any available storage media that can be accessed by a computer, and include both volatile and non-volatile media, removable media and non-removable media. By example, and not by limitation, computer-readable storage media or machine-readable storage media may be implemented in relation to any method or technique for storing information such as computer-readable or machine-readable instructions, program modules, structured data or unstructured data.

[0173] Computer-readable storage media are not limited to random access memory (RAM) and read-only memory (read Only memory (ROM), electrically erasable programmable read-only memory. This may include erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray disc (BD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible or non-temporary media that can be used to store desired information. In this regard, the terms “tangible” or “non-temporary” as applied herein to storage, memory, or computer-readable media should be understood to exclude only the transient signals themselves as modifiers, and not to waive any rights to all standard storage, memory, or computer-readable media that do not consist solely of transient signals themselves.

[0174] Computer-readable storage media can be accessed by one or more local or remote computing devices for various operations relating to the information stored in the media, for example, via access requests, queries, or other data retrieval protocols.

[0175] Communication media typically include any information distribution or transmission medium that embodies computer-readable instructions, data structures, program modules, or other structured or unstructured data in the form of data signals, such as modulated data signals, such as carrier waves or other transmission mechanisms. The term "modulated data signal" or signal refers to a signal in which one or more characteristics are set or modified in a manner that encodes information into one or more signals. Communication media include, but are not limited to, wired media such as wired networks or direct wired connections, as well as wireless media such as acoustic, RF, infrared, and other wireless media.

[0176] Referring again to Figure 18, an exemplary environment 1800 for implementing various embodiments of the aspects described herein includes a computer 1802, which includes a processing unit 1804, system memory 1806, and a system bus 1808. The system bus 1808 connects system components, including but not limited to the system memory 1806, to the processing unit 1804. The processing unit 1804 may be any of various commercially available processors. Dual microprocessors and other multiprocessor architectures can also be used as the processing unit 1804.

[0177] The system bus 1808 may be one of several types of bus structures that can further interconnect to a memory bus (with or without a memory controller), peripheral bus, and local bus using any of various commercially available bus architectures. The system memory 1806 includes ROM 1810 and RAM 1812. The basic input / output system (BIOS) may be stored in non-volatile memory such as ROM, erasable programmable read-only memory (EPROM), or EEPROM, and the BIOS includes basic routines that help transfer information between elements within the computer 1802, such as during startup. RAM 1812 may also include high-speed RAM, such as static RAM, for caching data.

[0178] Computer 1802 further includes an internal hard disk drive (HDD) 1814 (e.g., EIDE, SATA), one or more external storage devices 1816 (e.g., magnetic floppy disk drive (FDD) 1816, memory stick or flash drive reader, memory card reader, etc.), and drives 1820 that can read from or write to disks 1822 such as CD-ROMs, DVDs, BDs, etc., such as solid-state drives, optical disc drives, etc. Alternatively, if a solid-state drive is included, disks 1822 are not included unless they are separate. Although the internal HDD 1814 is illustrated as being located within computer 1802, the internal HDD 1814 may also be configured for external use in a suitable chassis (not shown). In addition, although not shown in environment 1800, a solid-state drive (SSD) may be used in addition to or instead of the HDD 1814. HDD1814, external storage device 1816, and drive 1820 can be connected to the system bus 1808 by HDD interface 1824, external storage interface 1826, and drive interface 1828, respectively. Interface 1824 for external drive implementation configurations is Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEJ). This may include at least one or both of the interface technologies of Electrical and Electronics Engineers (IEEE) 1394. Other external drive connection technologies are within the scope of the embodiments described herein.

[0179] Drives and their associated computer-readable storage media provide non-volatile storage such as data, data structures, and computer-executable instructions. In the case of computer 1802, drives and storage media accommodate the storage of any data in a suitable digital format. While the above description of computer-readable storage media refers to each type of storage device, it should be understood by those skilled in the art that other types of computer-readable storage media, whether currently existing or to be developed in the future, may also be used in exemplary operating environments, and furthermore, any such storage media may contain computer-executable instructions for carrying out the methods described herein.

[0180] Some program modules may be stored in a drive and RAM 1812, including an operating system 1830, one or more application programs 1832, other program modules 1834, and program data 1836. All or part of the operating system, applications, modules, or data may also be cached in RAM 1812. The systems and methods described herein can be implemented using various commercially available operating systems or combinations of operating systems.

[0181] Computer 1802 may optionally include emulation techniques. For example, a hypervisor (not shown) or other intermediary may emulate the hardware environment of operating system 1830, and the emulated hardware may optionally differ from the hardware illustrated in Figure 18. In such embodiments, operating system 1830 may include one VM among several virtual machines (VMs) hosted on computer 1802. Furthermore, operating system 1830 may provide a runtime environment for application 1832, such as a Java runtime environment or a .NET framework. The runtime environment is a consistent execution environment that enables application 1832 to run on any operating system that includes the runtime environment. Similarly, operating system 1830 may support containers, and application 1832 may be in the form of a container, which is a lightweight, standalone executable package of software containing, for example, code, runtime, system tools, system libraries, and configuration for the application.

[0182] Furthermore, computer 1802 can be made possible by using security modules such as a trusted processing module (TPM). For example, using a TPM, a boot component hashs the next boot component in time and waits for the result to match a secure value before loading the next boot component. This process can be performed at any layer in the code execution stack of computer 1802, for example, at the application execution level or the operating system (OS) kernel level, thereby enabling security at any level of code execution.

[0183] The user can input commands and information to the computer 1802 through one or more wired / wireless input devices, such as a keyboard 1838, a touchscreen 1840, and a pointing device such as a mouse 1842. Other input devices (not shown) may include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control, or other remote control, a joystick, a virtual reality controller or virtual reality headset, a gamepad, a stylus pen, an image input device such as a camera, a gesture sensor input device, a visual-motion sensor input device, an emotion or face detection device, or a biometric input device such as a fingerprint or iris scanner. These and other input devices are often connected to the processing unit 1804 via an input device interface 1844 which can be coupled to the system bus 1808, but can also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, or a BLUETOOTH® interface.

[0184] Monitor 1846 or other types of display devices may also be connected to the system bus 1808 via an interface such as a video adapter 1848. In addition to the monitor 1846, the computer typically includes other peripheral output devices (not shown), such as speakers and printers.

[0185] Computer 1802 can operate in a networked environment using logic connections via wired or wireless communication to one or more remote computers, such as remote computers 1850. Remote computers 1850 may be workstations, server computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, typically containing many or all of the elements described for computer 1802, but for brevity, only memory / storage devices 1852 are illustrated. The described logic connections include wired / wireless connections to local area networks (LANs) 1854 or larger networks, such as wide area networks (WANs) 1856. Such LAN and WAN networking environments are common in offices and businesses, facilitating enterprise-scale computer networks such as intranets, all of which can connect to global communication networks, such as the Internet.

[0186] When used in a LAN networking environment, computer 1802 may be connected to local network 1854 via a wired or wireless network interface or adapter 1858. Adapter 1858 may facilitate wired or wireless communication to LAN 1854, and LAN may also include a wireless access point (AP) placed on it to communicate with adapter 1858 in wireless mode.

[0187] When used in a WAN networking environment, computer 1802 may include a modem 1860, or it may connect to a communication server on the WAN 1856 via other means for establishing communication over the WAN 1856, such as the Internet. The modem 1860 may be internal or external, and wired or wireless, and may connect to the system bus 1808 via an input device interface 1844. In a networked environment, program modules drawn on computer 1802 or a part thereof may be stored in a remote memory / storage device 1852. The network connection shown is an example, and it will be understood that other means for establishing communication links between computers may be used.

[0188] When used in either a LAN or WAN networking environment, computer 1802 can access a cloud storage system or other network-based storage system in addition to, or instead of, the external storage device 1816 described above, such as a network virtual machine that provides one or more modes of storing or processing information. In general, the connection between computer 1802 and the cloud storage system can be established via LAN 1854 or WAN 1856, for example, by adapter 1858 or modem 1860, respectively. When computer 1802 is connected to the associated cloud storage system, the external storage interface 1826 can manage the storage provided by the cloud storage system, similar to other types of external storage, with the help of adapter 1858 or modem 1860. For example, the external storage interface 1826 can be configured to provide access to cloud storage sources as if those sources were physically connected to computer 1802.

[0189] Computer 1802 may be capable of communicating with any wireless device or entity configured to operate in wireless communication, such as a printer, scanner, desktop or portable computer, portable data assistant, communications satellite, any equipment or location associated with a wirelessly discoverable tag (e.g., kiosk, newsstand, merchandise shelf, etc.), and telephone. This may include Wireless Fidelity (Wi-Fi) and Bluetooth® wireless technologies. Thus, the communication may be a predefined structure, similar to conventional networks, or simply ad-hoc communication between at least two devices.

[0190] Figure 19 is a schematic block diagram of a sample computing environment 1900 in which the disclosed subject matter can interact. The sample computing environment 1900 includes one or more clients 1910. Client 1910 may be hardware or software (e.g., threads, processes, computing devices). The sample computing environment 1900 also includes one or more servers 1930. Server 1930 may also be hardware or software (e.g., threads, processes, computing devices). Server 1930 may house threads for performing transformations by using one or more embodiments such as those described herein. One possible communication between client 1910 and server 1930 may be in the form of data packets adapted for transmission between two or more computer processes. The sample computing environment 1900 includes a communication framework 1950 which may be used to facilitate communication between client 1910 and server 1930. Client 1910 is operably connected to one or more client data stores 1920 which may be used to store information local to client 1910. Similarly, server 1930 is operablely connected to one or more server datastores 1940 that may be used to store information local to server 1930.

[0191] Figure 20 shows exemplary and non-limiting apparatus for carrying out the various embodiments described herein. Figure 20 illustrates a non-limiting example of a dual-beam system 2010 having a vertically mounted scanning electron microscope (SEM) column and a focused ion beam (FIB) column mounted at an angle of about 52 degrees from the vertical. Such dual-beam systems are commercially available, for example, from FEI Company in Hillsboro, Oregon, the assignee of this application. Figure 20 shows an example of suitable microscopy hardware on which the various embodiments described herein may be implemented, but it should be noted that such microscopy hardware is non-limiting. In other words, the various embodiments described herein may be implemented in conjunction with any other suitable type of microscopy hardware. The dual-beam system 2010 is a non-limiting example of scientific instrument 302 or any other scientific instrument considered above.

[0192] The dual-beam system 2010 may include a scanning electron microscope 2041 together with a power supply and control unit 2045. By applying a voltage between the cathode 2052 and the anode 2054, an electron beam 2043 can be emitted from the cathode 2052. The electron beam 2043 can be focused into a fine spot by a condenser lens 2056 and an objective lens 2058. The electron beam 2043 can be scanned two-dimensionally over any suitable sample by a deflection coil 2060. The operation of the condenser lens 2056, the objective lens 2058, and the deflection coil 2060 can be controlled by the power supply and control unit 2045.

[0193] The electron beam 2043 can be focused onto a substrate 2022 which may be on the movable XY stage 2025 in the lower chamber 2026. When electrons in the electron beam 2043 collide with the substrate 2022, secondary electrons may be emitted. These secondary electrons can be detected by the secondary electron detector 2040, as discussed below. A transmission electron microscope (TEM) sample holder 2024 and a scanning transmission electron microscope (STEM) detector 2062 located below the movable XY stage 2025 can collect electrons passing through the sample mounted on the TEM sample holder 2024, as discussed above.

[0194] The dual-beam system 2010 may also include a focused ion beam (FIB) system 2011, which may comprise a vacuum chamber having an upper neck portion 2012, within which an ion source 2014 and a focusing column 2016 including an extraction electrode and an electrostatic-optical system may be located. The axis of the focusing column 2016 may be tilted 52 degrees (or any other preferred angular displacement) from the axis of the electron column. The ion column 2012 may include an ion source 2014, an extraction electrode 2015, a focusing element 2017, a deflection element 2020, and a focused ion beam 2018. A focused ion beam 2018 passes from an ion source 2014 through a focusing column 2016 and towards a substrate 2022, passing between electrostatic deflection means schematically indicated by the symbol 2020, the substrate 2022 may comprise, for example, a semiconductor device positioned on a movable XY stage 2025 in a lower chamber 2026.

[0195] The movable XY stage 2025 can move horizontally (along the X and Y axes) and vertically (along the Z axis). The movable XY stage 2025 can tilt about 60 degrees and rotate about the Z axis. In some embodiments, a separate TEM sample stage (not shown) may be used. Such a TEM sample stage may be movable along the X, Y, and Z axes. The door 2061 can be opened to insert the substrate 2022 onto the movable XY stage 2025, or to supply an internal gas supply reservoir, if one is used. The door 2061 can be interlocked so that it cannot be opened when the system is under vacuum.

[0196] An ion pump 2068 may be used to vacuum the neck portion 2012. Chamber 2026 may be vacuumed using a turbomolecular and mechanical pumping system 2030 under the control of a vacuum controller 2032. Such a vacuum system can fill chamber 2026 with approximately 1 × 10⁻¹⁶ units. -7 Tor ~ 5 x 10 -4 A vacuum of Torr can be provided. When etching aid gas, etching delay gas, or deposition precursor gas is used, the background pressure of the chamber is typically about 1 × 10⁻⁶. -5 It may rise to Tor.

[0197] The high-voltage power supply 2034 can provide an appropriate acceleration voltage to the electrodes in the focusing column 2016 to energize the focused ion beam 2018. When the focused ion beam 2018 collides with the substrate 2022, the material can be sputtered (i.e., physically ejected) from the sample. Alternatively, the focused ion beam 2018 can decompose a precursor gas to deposit the material.

[0198] A high-voltage power supply 2034 can be connected to an ion source 2014 (which may be a liquid metal ion source) and a suitable electrode in an ion beam focusing column 2016 to form an ion beam 2018 of approximately 1 keV to 60 keV and to direct the ion beam 2018 toward the sample. A deflection controller and amplifier 2036, operating according to a predetermined pattern provided by a pattern generator 2038, is coupled to a deflection element 2020 (which may be a deflection plate), thereby allowing the ion beam 2018 to track the corresponding pattern on the upper surface of the substrate 2022, either manually or automatically. In some systems, the deflection element 2020 may be positioned in front of the final lens. A beam blanking electrode (not shown) in the ion beam focusing column 2016 can cause the focused ion beam 2018 to collide with a blanking aperture (not shown) instead of the substrate 2022 when a blanking controller (not shown) applies a blanking voltage to the blanking electrode.

[0199] Ion source 2014 can provide, for example, a gallium metallic ion beam. In another example, ion source 2014 may be a plasma ion source that extracts ions from the generated plasma. The source can be focused into a beam of sub-1 / 10 micrometer width on the substrate 2022 for either modifying the substrate 2022 by ion milling, enhanced etching, or material deposition, or for the purpose of imaging the substrate 2022.

[0200] A charged particle detector 2040, such as an Everhart Thornley or multi-channel plate, used to detect secondary ions or electron emission, may be connected to a video circuit 2042 that can supply a drive signal to a video monitor 2044 and receive a deflection signal from a system controller 2019. The position of the charged particle detector 2040 within the lower chamber 2026 may vary in different embodiments. For example, the charged particle detector 2040 may be coaxial with the ion beam and may include holes for allowing the ion beam to pass through. In other embodiments, secondary particles may be collected through a final lens and then deflected from the axis for collection.

[0201] The micromanipulator 2047 can precisely move an object within a vacuum chamber. The micromanipulator 2047 may be equipped with a precision electric motor 2048 positioned outside the vacuum chamber to provide X, Y, Z, and theta control of a portion 2049 positioned within the vacuum chamber. The micromanipulator 2047 can be mated with different end effectors for manipulating small objects. In the various embodiments described herein, the end effector may be a slender probe 2050.

[0202] The gas delivery system 2046 may extend within the lower chamber 2026 to introduce and direct gas vapor toward the substrate 2022. U.S. Patent No. 5,851,413 by Casella et al., “Gas Delivery Systems for Particle Beam Processing,” assigned to the assignee of the present invention, describes a suitable gas delivery system 2046. Another gas delivery system is described in U.S. Patent No. 5,435,850 by Rasmussen, “Gas Injection System,” also assigned to the assignee of the present invention. For example, iodine may be delivered to enhance etching, or a metal-organic compound may be delivered to deposit metal.

[0203] The system controller 2019 can control the operation of various parts of the dual-beam system 2010. Through the system controller 2019, the user can cause the focused ion beam 2018 or electron beam 2043 to be scanned in a desired manner through commands entered into any preferred user interface (not shown). Alternatively, the system controller 2019 may control the dual-beam system 2010 according to programmed instructions stored in memory 2021. In various embodiments, any one of one or more software components 319 may be implemented in or otherwise executed by the system controller 2019.

[0204] Various embodiments may be systems, methods, apparatus, or computer program products in integration at any possible level of technical detail. A computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of various embodiments. The computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination thereof. A non-exhaustive list of more specific examples of computer-readable storage mediums may also include portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or grooved raised structures on which instructions are recorded, and any preferred combination thereof. The computer-readable storage media used herein should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0205] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers them for storage on a computer-readable storage medium within each computing / processing device. Computer-readable program instructions for performing the operations of various embodiments may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the C programming language or similar programming languages. Computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can connect to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or it can connect to an external computer (for example, via the Internet using an Internet service provider).In some embodiments, for example, electronic circuits including programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) can execute computer-readable program instructions by personalizing the electronic circuit using state information of computer-readable program instructions to implement various embodiments.

[0206] Various embodiments are described herein with reference to flowcharts or block diagrams of methods, apparatus (systems), and computer program products according to various embodiments. It will be understood that each block in a flowchart or block diagram, and combinations of blocks within a flowchart or block diagram, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device so that a machine can be manufactured such that instructions executed via the processor of the computer or other programmable data processing device create means for implementing a function / operation specified in one or more blocks of the flowchart or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium on which the instructions are stored can be instructed to function in a particular manner so that the storage medium contains a product comprising instructions that implement a mode of function / operation specified in one or more blocks of the flowchart or block diagram. Computer-readable program instructions can also be loaded onto a computer, another programmable data processing device, or another device to generate a computer-implemented process in which instructions executed on a computer, another programmable device, or another device implement a set of operational actions on the computer, another programmable device, or other device, such that the instructions implement a function / action specified in one or more blocks of a flowchart or block diagram.

[0207] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products in various embodiments. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logic function. In some alternative implementations, the functions described in a block may be performed in an order different from that shown in the figure. For example, two consecutively shown blocks may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and any combination of blocks in a block diagram or flowchart, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or performs a combination of dedicated hardware and computer instructions.

[0208] While the subject matter has been described above in the general context of computer executable instructions for computer program products running on one or more computers, those skilled in the art will recognize that the disclosure can be implemented in combination with other program modules. Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Furthermore, those skilled in the art will understand that various embodiments can be implemented using single-processor or multi-processor computer systems, minicomputing devices, mainframe computers, and other computer system configurations, including computers, handheld computing devices (e.g., PDAs, telephones), microprocessor-based or programmable consumer or industrial electronic devices. The illustrated embodiments can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network. However, some, though not all, embodiments of the disclosure can be implemented on a standalone computer. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0209] As used in this application, terms such as “component,” “system,” “platform,” and “interface” may refer to or include computer-related entities or entities relating to operating machines having one or more specific functions. Entities disclosed herein may be hardware, a combination of hardware and software, software, or running software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. For illustrative purposes, both an application running on a server and the server itself may be components. One or more components may reside within a process or an execution thread, and components may be localized on one computer or distributed across two or more computers. In another example, each component may run from various computer-readable media storing various data structures. Components may communicate via local or remote processes, such as following signals that have one or more data packets (e.g., data from a local system, another component in a distributed system, or a component interacting with other systems via signals across a network such as the Internet). As another example, a component may be a device having a specific function provided by mechanical parts operated by electrical or electronic circuits, which are operated by software or firmware applications executed by a processor. In such a case, the processor may be inside or outside the device and may execute at least part of the software or firmware application. As yet another example, a component may be a device that provides a specific function through electronic parts without mechanical parts, and the electronic parts may include a processor or other means for executing software or firmware that gives at least part of the functionality of the electronic parts.In one embodiment, the component can emulate an electronic component, for example, via a virtual machine in a cloud computing system.

[0210] In addition, the term “or” is intended to mean an inclusive “or,” not an exclusive “or.” That is, unless otherwise specified or it is clear from the context, “X uses A or B” is intended to mean any natural inclusive substitution. That is, if X uses A, if X uses B, or if X uses both A and B, “X uses A or B” is satisfied under any of the aforementioned cases. Where used herein, the terms “and / or” are intended to have the same meaning as “or.” Furthermore, the articles “a” and “an” as used herein and in the accompanying drawings should generally be interpreted as meaning “one or more,” unless otherwise specified or it is clear from the context that they refer to the singular form. Where used herein, the terms “example” or “exemplary” are used to mean serving as an example, case, or illustration. To avoid misunderstanding, the subject matter disclosed herein is not limited by such examples. In addition, any embodiment or design described herein as “example” or “exemplary” should not necessarily be construed as being preferable or advantageous to other embodiments or designs, nor should it be meant to exclude equivalent exemplary structures and techniques known to those skilled in the art.

[0211] The disclosures herein describe non-limiting examples. For ease of description or explanation, various parts of the disclosures herein use the terms “each,” “all,” or “all” when considering various examples. Such use of the terms “each,” “all,” or “all” is non-limiting. In other words, when the disclosures herein provide descriptions that apply to “each,” “all,” or “all” of some particular objects or components, please understand that this is a non-limiting example, and further understand that in various other examples, such descriptions may apply to fewer than “each,” “all,” or “all” of those particular objects or components.

[0212] As used herein, the term “processor” can refer to substantially any computing unit or device, including, but not limited to, single-core processors, single processors with software multithreading capability, multi-core processors, multi-core processors with software multithreading capability, multi-core processors with hardware multithreading technology, and parallel platforms with distributed shared memory. In addition, a processor can refer to integrated circuits, application-specific integrated circuits (ASICs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic controllers (PLCs), complex programmable logic devices (CPLDs), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Furthermore, a processor may utilize nanoscale architectures, including, but not limited to, molecular and quantum dot-based transistors, switches, and gates, to optimize space use or improve the performance of user equipment. A processor can also be implemented as a combination of computing units. In this disclosure, terms such as “store,” “storage,” “datastore,” “data storage,” “database,” and substantially any other information storage component relating to the operation and functionality of a component are used to refer to a “memory component,” an entity embedded in “memory,” or a component that contains memory. It should be understood that the memory or memory component described herein may be either volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.As an example, not an limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random-access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). Volatile memory may include, for example, RAM that can function as external cache memory. As an example, not an limitation, RAM may include synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data-rate SDRAM (double). Data rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Direct Rambus RAM (direct Rambus RAM, DRRAM), Direct Rambus Dynamic RAM It is available in many forms, such as dynamic RAM (DRDRAM) and Rambus dynamic RAM (RDRAM). In addition, the memory components disclosed in the systems or computer implementations herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0213] The above descriptions include only examples of systems and computer implementations. Of course, it is impossible to describe all possible combinations of components or computer implementations for the purpose of illustrating this disclosure, but many further combinations and substitutions of the disclosure are possible. Furthermore, to the extent that terms such as “includes,” “has,” and “possesses” are used in the detailed description, claims, appendices, and drawings, such terms are intended to be as comprehensive as the term “comprising” is interpreted when the term “comprising” is used as a transitional term in the claims.

[0214] The descriptions of various embodiments are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the embodiments disclosed. Many modifications and variations will be apparent without departing from the scope and spirit of the embodiments described. The terminology used herein has been chosen to best describe the principles of the embodiments, their practical application to market-based technologies or technical improvements, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0215] Examples Various non-limiting embodiments are described in the following examples.

[0216] Example 1: The system may include a processor that runs computer executable components stored in non-temporary computer-readable memory, the computer executable components including a workflow component that causes a charged particle microscope to execute a workflow on a sample, a status component that can retrieve runtime data logged by the charged particle microscope during the workflow in response to the charged particle microscope generating an error message that interrupts the workflow, and a model component that can synthesize a first text explaining why the workflow was interrupted by running a large-scale language model on the error message and runtime data.

[0217] Example 2: The system of any prior example can be implemented, and the model component can further synthesize a second text describing how the workflow is resolved through the execution of a large-scale language model on the first text.

[0218] Example 3: The system of any prior example can be implemented, and the model component can further synthesize coding scripts configured to cause the charged particle microscope to resolve the workflow through the execution of a large-scale language model on a second text.

[0219] Example 4: The system of any prior embodiment can be implemented, and the computer executable component further includes an execution component that can cause the charged particle microscope to execute a coding script, thereby resolving the workflow.

[0220] Example 5: Any system from a prior example can be implemented, and the workflow may include imaging scanning of a sample, milling operations on a sample, or movement of a movable stage that holds the sample.

[0221] Example 6: The system of any prior example can be implemented, and the runtime data may include partial images of the sample captured by the charged particle microscope during the workflow, or the state of one or more configurable operating parameters of the charged particle microscope during the workflow.

[0222] Example 7: The system of any prior embodiment can be implemented, and a large-scale language model can synthesize a first text in a search-enhanced generation manner based on performing an embedded search through a repository of technical documents associated with the design, manufacture, operation, or troubleshooting of charged particle microscopes, and the computer executable component may further include an execution component that can receive feedback from users or technicians regarding the first text, and the model component can synthesize text edits to documents in the repository of technical documents retrieved by the embedded search through the execution of the large-scale language model in response to the feedback, and the execution component can update the repository of technical documents by inserting text edits into the documents.

[0223] Example 8: The system of any prior embodiment can be implemented, and the computer executable component may further include an executable component that can visually or audibly render the first text on an electronic screen or electronic speaker.

[0224] In various embodiments, any combination of Examples 1 to 8 can be implemented.

[0225] Example 9: A computer implementation may include: causing a charged particle microscope to perform a workflow on a sample using a device operably coupled to a processor; retrieving runtime data logged by the charged particle microscope during the workflow, in response to the device generating an error message that interrupts the workflow; and synthesizing a first text explaining why the workflow was interrupted, in response to the device running a large-scale language model on the error message and runtime data.

[0226] Example 10: A computer implementation of any prior embodiment can be implemented, further comprising synthesizing a second text describing how the workflow is resolved by a device and through the execution of a large-scale language model on a first text.

[0227] Example 11: A computer implementation of any prior embodiment can be implemented, further comprising synthesizing a coding script configured to cause a charged particle microscope to resolve a workflow via a device and through the execution of a large-scale language model on a second text.

[0228] Example 12: A computer implementation method of any prior embodiment can be implemented, further including having the device execute a coding script on a charged particle microscope, thereby resolving the workflow.

[0229] Example 13: A computer-aided implementation of any prior embodiment can be implemented, and the workflow may include imaging scanning of a sample, milling operations on a sample, or movement of an operable stage holding a sample.

[0230] Example 14: A computer implementation of any prior embodiment can be implemented, and the runtime data may include partial images of the sample captured by the charged particle microscope during the workflow, or states that one or more configurable operating parameters of the charged particle microscope had during the workflow.

[0231] Example 15: A computer implementation of any prior embodiment can be implemented, in which a large-scale language model can synthesize a first text in a search-enhanced generation manner based on performing an embedded search through a repository of technical documentation associated with the design, manufacture, operation, or troubleshooting of charged particle microscopes, the device further includes receiving feedback from a user or technician regarding the first text, the device synthesizing text edits to documents in the repository of technical documentation retrieved by the embedded search, and the device updating the repository of technical documentation by inserting text edits into the documents.

[0232] Example 16: A computer implementation of any prior embodiment can be implemented, further comprising rendering the first text visually or audibly on an electronic screen or electronic speaker by the device.

[0233] In various embodiments, any combination of Examples 9 to 16 can be implemented.

[0234] Example 17: A computer program product for facilitating large-scale language modeling of scientific instrument workflow interruptions may comprise non-temporary computer-readable memory into which program instructions are incorporated. In various embodiments, the program instructions may be made executable by the processor to cause the scientific instrument to perform a workflow on a sample, to cause the scientific instrument to generate an error message that interrupts the workflow, to retrieve runtime data logged by the scientific instrument during the workflow, and to synthesize a first text explaining why the workflow was interrupted through the execution of a large-scale language model on the error message and runtime data.

[0235] Example 18: A computer program product of any prior embodiment can be implemented, and the program instructions can be made executable to cause the processor to synthesize a second text describing how to resolve a workflow through the execution of a large-scale language model on a first text.

[0236] Example 19: A computer program product of any prior embodiment can be implemented, and the program instructions can further cause the processor to synthesize a coding script configured to cause a scientific instrument to resolve a workflow through the execution of a large-scale language model on a second text, and make executable so that the scientific instrument executes the coding script and thereby resolves the workflow.

[0237] Example 20: Any computer program product from a prior example can be implemented, and the scientific instrument can be a charged particle microscope, chromatograph, or mass spectrometer.

[0238] In various embodiments, any combination of Examples 17 to 20 can be implemented.

[0239] In various embodiments, any combination of Examples 1 to 20 can be implemented.

Claims

1. It is a system, A processor that executes computer executable components stored in non-temporary computer-readable memory, wherein the computer executable components A workflow component that performs a workflow on a sample in a charged particle microscope, A status component that searches runtime data logged by the charged particle microscope during the workflow in response to the charged particle microscope generating an error message that interrupts the workflow, A system including a model component that synthesizes a first text explaining why the workflow was interrupted, through the execution of a large language model on the error message and runtime data.

2. The system according to claim 1, wherein the model component further synthesizes a second text describing how the workflow is resolved through the execution of the large-scale language model on the first text.

3. The system according to claim 2, wherein the model component further synthesizes a coding script configured to cause the charged particle microscope to resolve the workflow through the execution of the large-scale language model on the second text.

4. The aforementioned computer executable component, The system according to claim 3, further comprising an execution component that causes the charged particle microscope to execute the coding script, thereby resolving the workflow.

5. The system according to claim 1, wherein the workflow includes imaging scanning of the sample, milling operations on the sample, or movement of an operable stage that holds the sample.

6. The system according to claim 1, wherein the runtime data includes partial images of the sample captured by the charged particle microscope during the workflow, or states that one or more configurable operating parameters of the charged particle microscope had during the workflow.

7. The large-scale language model synthesizes the first text in a search-enhanced generation manner based on performing an embedded search through a repository of technical documentation associated with the design, manufacture, operation, or troubleshooting of the charged particle microscope, and the computer executable component, The system according to claim 1, further comprising an execution component that receives feedback from a user or engineer regarding the first text, the model component synthesizes text edits to documents in the repository of technical documents retrieved by the embedded search via the execution of the large language model on the feedback, and the execution component updates the repository of technical documents by inserting the text edits into the documents.

8. The aforementioned computer executable component, The system according to claim 1, further comprising an execution component for visually or audibly rendering the first text on an electronic screen or an electronic speaker.

9. A computer implementation method, A device operablely coupled to the processor enables the charged particle microscope to perform a workflow on the sample, The device, in response to the charged particle microscope generating an error message that interrupts the workflow, searches for runtime data logged by the charged particle microscope during the workflow. A computer implementation method comprising synthesizing, by the device and through the execution of a large language model on the error message and runtime data, a first text explaining why the workflow was interrupted.

10. The computer implementation method according to claim 9, further comprising synthesizing a second text describing how the workflow is resolved by the device and through the execution of the large-scale language model on the first text.

11. The computer implementation method according to claim 10, further comprising synthesizing a coding script configured to cause the charged particle microscope to resolve the workflow via the device and through the execution of the large-scale language model on the second text.

12. The computer implementation method according to claim 11, further comprising using the device to cause the charged particle microscope to execute the coding script, thereby resolving the workflow.

13. The computer-aided method according to claim 9, wherein the workflow includes imaging scanning of the sample, milling operations on the sample, or movement of an operable stage holding the sample.

14. The computer-aided method according to claim 9, wherein the runtime data includes partial images of the sample captured by the charged particle microscope during the workflow, or states that one or more configurable operating parameters of the charged particle microscope had during the workflow.

15. Based on the large-scale language model performing an embedded search through a repository of technical documentation related to the design, manufacture, operation, or troubleshooting of the charged particle microscope, the first text is synthesized in a search-enhanced generation manner. The device receives feedback from a user or technician regarding the first text, The device, and through the execution of the large-scale language model on the feedback, synthesizes text edits to documents in the repository of technical documents retrieved by the embedded search, The computer implementation method according to claim 9, further comprising updating the repository of technical documents by inserting the text edits into the document using the device.

16. The computer implementation method according to claim 9, further comprising rendering the first text visually or audibly on an electronic screen or electronic speaker using the device.

17. A computer program product for facilitating the resolution of large-scale language models of scientific instrument workflow interruptions, wherein the computer program product comprises non-temporary computer-readable memory into which program instructions are embedded, and the program instructions are transmitted to a processor, Allow scientific instruments to perform a workflow on the sample. In response to the scientific instrument generating an error message that interrupts the workflow, the runtime data logged by the scientific instrument during the workflow is retrieved. A computer program product executable by the processor to synthesize a first text explaining why the workflow was interrupted, through the execution of a large-scale language model on the error message and runtime data.

18. The program instruction further sends to the processor: The computer program product according to claim 17, which is executable to cause a second text describing how to resolve the workflow through the execution of the large-scale language model on the first text.

19. The program instruction further sends to the processor: By running the large-scale language model on the second text, a coding script is synthesized that is configured to cause the scientific instrument to resolve the workflow. The computer program product according to claim 18, wherein the scientific instrument is executable to cause the scientific instrument to execute the coding script and thereby resolve the workflow.

20. The computer program product according to claim 17, wherein the scientific instrument is a charged particle microscope, a chromatograph, or a mass spectrometer.

Images