Digital sample model for instrument optimization
By simulating the interaction between a charged particle microscope and a sample using a digital twin system, a physical model is generated to optimize operating settings. This solves the problem of complex instruments requiring highly skilled personnel to operate, and achieves automated and efficient control.
Patent Information
- Application Number
- CN202580028614.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-17
- Filing Date
- 2025-04-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing charged particle microscope systems are complex and require highly skilled technicians to operate, limiting their use to users with varying levels of expertise.
By simulating the interaction between physical assets and samples using a digital twin system, a physical-based digital model is generated to reconstruct the chemical and physical properties of the samples. Based on the simulation results, operational settings are optimized to achieve automated control of the physical assets.
It reduces reliance on user expertise, improves operational efficiency and accuracy, and enables automated and precise control of charged particle microscope systems.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Cross-references to related applications
[0001] This application claims priority to U.S. Provisional Application No. 63 / 635,491, filed April 17, 2024, entitled “DIGITAL TWINS FOR SAMPLE-BEAMINTERACTIONS IN CHARGED PARTICLE MICROSCOPY,” the entire disclosure of which is incorporated herein by reference. Background Technology
[0002] Modern scientific instruments, such as charged particle microscopes, are typically extremely complex systems with a variety of interacting components. Many of these instruments can only be successfully operated and controlled by highly skilled technicians, thus limiting their usefulness in scientific and commercial settings. Therefore, there is a need for systems and methods that enable users of varying skill levels to operate these complex scientific instruments. Summary of the Invention
[0003] As described herein, in one embodiment, a scientific instrument support system includes: a sample; a physical asset configured to interact with the sample; and one or more computing devices storing executable instructions. When executed, the executable instructions cause the one or more computing devices to generate one or more predictions based on simulations of one or more of the sample, the physical asset, or the interaction between the physical asset and the sample. The executable instructions further cause the one or more computing devices to update a physics-based digital model of the sample based on the accuracy of the one or more predictions, wherein the physics-based digital model is used to reconstruct the chemical and physical properties of the sample to generate simulations of the sample.
[0004] In at least one embodiment, a method includes receiving a request to perform operations on a sample using a physical asset, and in response to the request, performing one or more simulations of the interaction between the physical asset and the sample, at least in part based on a digital model of the sample. The method also includes returning optimized settings for the physical asset based on the one or more simulations. The digital model is updated based on a comparison of the results using the optimized settings with predicted results.
[0005] In another embodiment, a non-transitory computer-readable medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform one or more simulations based on requested results, using a physical-based model of a sample and a physical-based model of a physical asset, and to predict, based on the one or more simulations, a setup to be used to operate the physical asset. The computer-executable instructions further cause the one or more processors to operate the physical asset using the predicted setup; compare the actual results of the operation of the physical asset with the requested results; and use the comparison to adjust the physical-based model of the sample. Attached Figure Description
[0006] To facilitate the identification of any particular element or action in the discussion, one or more of the largest significant digits in the reference numerals refer to the drawing number in which the element is first introduced.
[0007] Figure 1 An example is illustrated of a computing system comprising communication coupling between a physical asset controller and a digital twin system according to one implementation scheme.
[0008] Figure 2 An example is shown of a set of component models that may be included in a digital twin system according to one implementation scheme.
[0009] Figure 3 An example is illustrated of a charged particle dual-beam microscope system that can be simulated by a digital twin system according to one embodiment.
[0010] Figure 4A An image generated from physical assets according to one implementation scheme is shown.
[0011] Figure 4B An example is given of a digital twin system based on one implementation scheme. Figure 4A Image analysis.
[0012] Figures 5A to 5D An example of a simulation of a milling plan using a digital twin system based on one implementation is presented.
[0013] Figure 6 An example of a system for simulating sample milling is shown according to one implementation.
[0014] Figure 7 An example is given of a specific implementation of artificial intelligence in a digital twin system according to one implementation plan.
[0015] Figure 8 An example is provided of a physical asset support module for performing supporting operations according to one implementation scheme.
[0016] Figure 9An example is given of a method for performing support operations on physical assets according to one implementation scheme.
[0017] Figure 10 An example is given of a method for optimizing the operation of physical assets by using a digital twin system in conjunction with a physical asset according to an implementation plan.
[0018] Figure 11 An example is given of a method for updating a digital twin system according to one implementation plan.
[0019] Figure 12 An example is given of a method for training an AI-based model according to an implementation plan.
[0020] Figure 13 A graphical user interface according to one embodiment is illustrated, which can be used to facilitate the execution of at least some of the supporting operations for the physical assets described herein.
[0021] Figure 14 A computing device is illustrated according to one embodiment, which can implement at least some of the supporting methods described herein.
[0022] Figure 15 An example is provided of a physical asset support system that is executable for at least some of the supporting operations of the physical assets described herein, according to one implementation scheme. Detailed Implementation
[0023] This document discloses physical asset (e.g., scientific instrument) support systems, as well as related methods, computing devices, and computer-readable media. For example, in some embodiments, the systems and methods disclosed herein can provide physics-based simulations of sample-beam interactions in charged particle microscopy, thereby enabling improved simulation and control of complex microscopes. For example, sample-beam interactions may include the impact of charged particle beams on a sample and their effects. As described herein, physics-based simulations can be used with physics-based digital models, where "physics-based" means using physical principles and laws. Physics-based digital models can be used to reconstruct, for example, the chemical and physical properties of a sample to be analyzed, measured, or otherwise interacted with by a physical asset. A set of physics-based simulations representing the operation and control of a particular physical asset may also be referred to hereinafter as a digital twin of the scientific instrument, and a set of physics-based simulations representing a sample may also be referred to hereinafter as a digital twin of the sample. Simulations can be used to support users of various scientific instrument systems, where users may have different experiences and expertise regarding the operation of the scientific instrument. Information generated from simulations can guide users in the operation and control of the scientific instrument without requiring a high degree of prior knowledge, thereby reducing the time and effort required for users to be able to use the functionality of the scientific instrument.
[0024] The scientific instrument support implementations disclosed herein can achieve improved performance compared to conventional methods. For example, various embodiments of the disclosed implementations enable complete simulation and control of charged particle microscopy and its interaction with samples (e.g., during imaging, milling, or other interactions). Thus, the disclosed implementations provide improvements to scientific instrument technology (e.g., improvements in the computer technology supporting such scientific instruments, and others).
[0025] The various embodiments disclosed herein improve upon conventional methods to achieve faster and more accurate simulation and control through more complete modeling. Such technical advantages cannot be achieved through conventional and traditional methods, and all users of systems incorporating such embodiments can benefit from them. Therefore, the technical features of the embodiments disclosed herein are clearly unconventional in the field of scientific instruments, as are the combinations of features of the embodiments disclosed herein. As further discussed herein, various aspects of the embodiments disclosed herein can improve the functionality of the computer itself; for example, a computer supporting the operation of a charged particle microscope. The computational and user interface features disclosed herein involve not only the collection and comparison of information but also the application of new analytical and technical means to alter the operation of the charged particle microscope system. Therefore, this disclosure introduces functionality that is impossible for conventional computing devices and humans to perform.
[0026] Therefore, embodiments of this disclosure can serve any of many technical purposes, such as controlling a particular technical system or process (e.g., charged particle microscopy); determining how to control a machine based on measurement results; and simulating the behavior of a technical project or process (e.g., charged particle microscopy systems and processes). Specifically, this disclosure provides technical solutions to technical problems, including but not limited to the challenges of simulating and controlling complex scientific instruments when interacting with samples.
[0027] Therefore, the embodiments disclosed herein provide improvements to scientific instrument technology (e.g., improvements to computer technology supporting scientific instruments, and other improvements).
[0028] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description, wherein like reference numerals always indicate like parts, and practical embodiments are shown in the drawings by way of illustration. It should be understood that other embodiments may be utilized, and structural or logical changes may be made, without departing from the scope of this disclosure. Therefore, the following detailed description should not be regarded as limiting.
[0029] The various operations can be described sequentially as multiple discrete actions or operations in a manner most conducive to understanding the subject matter disclosed herein. However, the described order should not be construed as implying that these operations must depend on the order. Specifically, these operations may not be performed in the order presented. The described operations may be performed in a different order than the described embodiments. Various additional operations may be performed, and / or the described operations may be omitted in additional embodiments.
[0030] For the purposes of this disclosure, the phrases “A and / or B” and “A or B” mean (A), (B), or (A and B). For the purposes of this disclosure, the phrases “A, B and / or C” and “A, B or C” mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B and C). Although some elements may be represented in the singular (e.g., “processing device”), any suitable element may be represented by multiple instances of that element, and vice versa. For example, a set of operations described as being performed by a processing device may be implemented as different operations of those operations performed by different processing devices. As used herein, the phrase “based on” should be understood to mean “at least partially based on”, unless otherwise specified.
[0031] This specification uses the phrases "implementation," "various embodiments," and "some embodiments," each of which can refer to one or more embodiments of the same or different implementations. Furthermore, the terms "comprising," "including," "having," etc., used with respect to embodiments of this disclosure are synonymous. When used to describe a range of dimensions, the phrase "between X and Y" indicates a range including both X and Y. As used herein, "apparatus" can refer to any single device, a collection of devices, a part of a device, or a collection of parts of a device. The drawings are not necessarily drawn to scale.
[0032] Figure 1 This is a diagram of a computing system 100 in which physical assets 102 (such as a charged particle microscope 106 interacting with sample 108) and a digital twin system 104 simulating physical assets 102 are communicatively coupled (e.g., as indicated by the dashed arrows). Physical asset 102 can be one of various types of charged particle microscopes, including but not limited to scanning electron microscopy (SEM) or transmission electron microscopy (TEM). Digital twin system 104 includes physical-based digital replicas of physical assets 102 and physical sample 108 (e.g., by a digital twin system simulating physical assets 102). Figure 1The examples illustrate the synthetic physical asset 110 and synthetic physical sample 112, as well as the interaction between physical asset 102 and physical sample 108. In one example, digital twin system 104 is implemented at computing device 114, which may be coupled to the control system of physical asset 102 via a wireless or wired connection. In another example, computing device 114 may host both digital twin system 104 and the control system of physical asset 102. Digital twin system 104 may include one or more digital twins (e.g., physics-based simulators) to predict each of the following: the behavior of physical asset 102, the properties of physical sample 108, and the outcome of the interaction between physical asset 102 and physical sample 108.
[0033] This coupling allows digital twin system 104 to generate synthetic image 116 that simulates an instrument image of physical sample 108 generated by physical asset 102. Therefore, a digital twin system (e.g., including one or more digital twins) can construct a virtual representation of the system using a physically-based model, incorporating a complete image of the physical asset. It should be understood that although digital twin system 104 is described in the context of image generation, it can be used to simulate the generation of various types of data other than images, as well as processes and operations performed by the physical asset, and the behavior of the physical sample in response to these processes and operations.
[0034] Furthermore, in at least one implementation, artificial intelligence techniques such as machine learning, deep learning, natural language processing, and computer vision can be used to implement the digital twin system. In other examples, models of real-world systems, such as physical assets and / or materials, and suitable programming languages, algorithms, and software can be used to implement the digital twin system. Regardless of the implementation mode, the digital twin system can be similarly verified, validated, and updated in a continuous and ongoing manner.
[0035] Figure 2 An example set of component models 200 is illustrated that may be included in one or more digital twins in a digital twin system. Component models 200 include computing devices (such as...) Figure 1 Computing device 114 or Figure 14 Software code (e.g., algorithms) implemented at a computing device 1400. The software code is configured to simulate, for example, Figure 1 The physical asset 102 comprises components of the physical asset, and data processing is performed at the control system of the physical asset. For example, the physical asset may be a charged particle microscope. In at least one embodiment, the component model 200 allows simulation of a charged particle microscope and the results generated by the charged particle microscope.
[0036] like Figure 2As shown, this set of example component models 200 includes a source model 202 simulating a charged particle source (e.g., an electron beam source or an ion beam source), a beam optics model 204 simulating the manipulation of a charged particle beam generated by the charged particle source, a probe formation model 206 simulating the generation of a focused beam of charged particles, and a scanning model 208 simulating scanning a focused beam across a sample. This set of example component models 200 also includes: a detector model 210 simulating the capture of signals generated at a detector on a physical asset via the interaction of the focused beam with the sample (e.g., secondary electron detection, backscattered electron detection, X-ray detection, transmission electron detection, etc.); a signal receiving model 212 simulating the capture and processing of signals by a detector for use as image data; a beam-sample interaction model 214 simulating reactions and activities occurring at the sample and the effects on the charged particle beam caused by the interaction between the beam and the sample; and a sample definition model 216 simulating the visualization and differentiation of features from image data. The set of example component models 200 also includes: an acquisition model 218 that simulates the collection and recording of image data; a raw image model 220 that simulates the generation of an initial unprocessed image captured by a detector upon receiving a signal; a digital image post-processing model 222 that simulates applying various techniques to the initial image to enhance image quality, correct artifacts, and / or extract information from it; and a synthetic image model 224 that simulates the generation of a final processed image output by the control system of a charged particle microscope.
[0037] Therefore, computing systems including digital twin systems utilizing this set of component models 200 (e.g., Figure 1 The computational system 100 can represent a complete feedback loop between the digital twin system and physical assets (including, for example, tools). As described above, in one example, such a computational system can generate artificial images for specific samples and operating conditions. In some embodiments, the digital twin system can operate in parallel with the physical assets, and the system can use this parallel operation to simulate, examine, tune, or otherwise adjust the operation of the physical assets to achieve higher accuracy, fewer problems requiring human technician intervention, and improved image quality. In other words, the digital twin can be used to continuously update and optimize the operating settings of the physical assets to produce the desired results when the physical assets are applied to the sample. This is achieved by continuously updating the model of the sample using feedback from the physical assets. In at least one embodiment, feedback can be acquired in real time and automatically to update and improve the accuracy of the model simulating the sample. Then, when operations are subsequently performed on the same type of sample, optimized settings can be inferred and recommended.
[0038] In some implementations, a digital twin system may include a model (e.g., a digital twin) of a physical asset (e.g., a scientific instrument) and a model of a sample that interacts with the physical asset. The digital twin of the physical asset may be generated at least in part based on a representation of the physical asset (e.g., computer-aided design (CAD) drawings). For example, as... Figure 3 As shown, the physical asset can be a charged particle dual-beam microscope system 300 (such as a focused ion beam scanning electron microscope system or FIB-SEM) including an electron source 302. The electron source 302 generates an electron beam that propagates through an electron beam column 304.
[0039] Electron source 302 may include one or more emitters configured to generate and direct electrons into electron beam column 304. These emitters may include thermionic emitters, Schottky emitters, field emission source emitters, or combinations thereof, operatively coupled to a power system configured to apply a high voltage (e.g., on the order of kilovolts to hundreds of kilovolts) to the emission region of the emitter material. Electron beam column 304 includes electromagnetic optics (e.g., electrostatic lenses, electromagnetic lenses, monochromators, etc.) and apertures configured to shape, focus, defocus, and guide the electron beam according to a set of operating parameters, such that the electron beam is focused onto sample 308 along beam axis A. These operating parameters may include beam current, beam energy (e.g., in volts, electron volts, etc.), magnification parameters, scanning pattern, dwell time, and / or one or more pulse parameters. In this way, the example system 300 can be used as a SEM to partially image the sample 308 and / or can be used to deposit material onto the sample 308 with electron beam assistance (e.g., in conjunction with a gas injection system or in coordination with GIS 318).
[0040] Sample 308 is located within the vacuum chamber 312 of the charged particle dual-beam microscope system 300, which allows sample 308 to be maintained under reduced pressure (e.g., below atmospheric pressure). The position of the sample relative to the electron beam column 304 can be adjusted using the stage controller 306.
[0041] The charged particle dual-beam microscope system 300 may further include an ion source 314 that generates an ion beam propagating through the FIB column 316. The ion source 314 may include one or more components configured to generate the ion beam and guide the ions along beam axis B into the FIB column 316. Similar to the electron beam column 304, the FIB column may include electromagnetic optics (e.g., electrostatic lenses, electromagnetic lenses, monochromators, etc.) and an aperture configured to shape, focus, defocus, and guide the ion beam according to a set of operating parameters, such that the ion beam is focused onto the sample 308. These operating parameters may include beam current, beam energy (e.g., in volts, electron volts, etc.), magnification parameters, scanning pattern, dwell time, and / or one or more pulse parameters. In this way, the example system 300 can be used as an FIB to remove portions of the sample 308 and / or can be used to ion-beam-assisted deposition of material onto the sample 308 (e.g., in coordination with GIS 318).
[0042] The charged particle dual-beam microscope system 300 may also include one or more retractable devices (such as GIS 318). GIS 318 may be oriented toward sample 308 and configured to direct gas flow into vacuum chamber 312. The gas introduced into vacuum chamber 312 by gas injection system 318 may be used to etch sample 308, deposit material onto sample 308, enhance imaging contrast or resolution, and / or modify the surface properties of sample 308.
[0043] Control circuitry (e.g., controller 301) can coordinate the operation of one or more components of the example system 300 based on machine-executable instructions (e.g., software, firmware, etc.) received from an external system, which may be stored in a machine-readable storage medium and / or via wired and / or wireless communication technologies (e.g., via WiFi or Bluetooth links). For this purpose, components of the example system can be automated (e.g., operating without human intervention), pseudo-automated (e.g., operating with limited human intervention to initiate operations, analyze outputs, and confirm, etc.), or manually operated (e.g., where individual operations of the example system 300 are performed by a human user). In one exemplary example, sample stage 310 may be mechanically coupled to an automated stage controller 306, allowing sample 308 to be reversibly tilted relative to beam axes A and B, such that the surface of sample 308 is substantially normal to the given beam axis during operation of the corresponding charged particle beam source.
[0044] Controller 301 can be configured to be similar to Figure 14The computing device 1400 is depicted and further described below. In at least one embodiment, a controller 301 receives signals from various sensors and detectors of the charged particle dual-beam microscope system 300 and sends instructions to actuators (e.g., actuable components) of the charged particle dual-beam microscope system 300 based on the signals. Furthermore, the controller 301 may communicatively couple to a digital twin system (e.g., such as...). Figure 1 As shown, information such as settings and / or parameters for the operation of the charged particle dual-beam microscope system 300 can be transmitted from the digital twin system to the controller 301, so that the controller 301 applies this information to the operation of the charged particle dual-beam microscope system 300. Furthermore, the digital twin system can obtain experimental results from the controller 301 to update the sample model of the digital twin system, at least based on these results. In some cases, the digital twin system can control the operation of the charged particle dual-beam microscope system 300 through communication coupling between the controller 301 and the digital twin system.
[0045] Embodiments of this disclosure may omit one or more components of the example system 100. For example, one or more of sources 302 and 314 and / or pillars 304 and 316 may be omitted. In one exemplary example, the SEM system may be configured to perform the beam-induced deposition process of this disclosure.
[0046] In at least one embodiment, as described above, CAD drawings of at least a portion of the physical assets, such as the charged particle dual-beam microscope system 300, can be provided to a digital twin system (e.g., input to a computing device implementing the digital twin system) for modeling or simulation by the digital twin system. For example, the digital twin of the physical assets of the digital twin system can be used to model the behavior and / or operation of the charged particle dual-beam microscope system 300 to predict the state of the charged particle dual-beam microscope system 300 and / or control the charged particle dual-beam microscope system 300. The CAD drawings may, for example, include the entire charged particle dual-beam microscope system 300 or a portion thereof, and any other pillars or devices coupled to the vacuum chamber 312. In other examples, multiple CAD drawings of different portions of the electron microscope system 300 can be used to model the operation of the charged particle dual-beam microscope system 300 via a digital twin system. Furthermore, in some examples, additional information about the physical assets (e.g., instrument specifications and parameters) can be provided with one or more CAD drawings. Figure 1The data is input into a computing device to allow the digital twin to accurately model the physical asset. The digital twin can further obtain information about the charged particle dual-beam microscope system 300 (such as device properties and characteristics) via communicative coupling between the digital twin system and the controller 301 of the charged particle dual-beam microscope 300. The control system can acquire data from various databases and other data repositories or archives. The acquired data can be used, for example, to populate one or more lookup tables to generate recommended operating parameters for the physical asset.
[0047] In a digital twin system, a digital twin of a sample can model the physical and chemical properties of the sample and can be used, for example, to predict the effects of physical assets on the sample (e.g., predicting the effects of charged particle beams from a charged particle microscope on the sample). For example, a sample digital twin can be used in conjunction with a physical asset digital twin and / or a digital twin that models the interaction between the physical asset and the sample to obtain optimized instrument settings for operating the physical asset relative to a requested operation. As an example, the physical asset can be used to perform operations based on sample information such as that incorporated into the physical asset digital twin (e.g., sample type, identification, material, etc.) and the characteristics of the physical asset. Sample information can be input by, for example, an operator, or retrieved from a database or lookup table based on operator selection, but other ways of providing sample information to the digital twin system are also possible. The sample digital twin can be initially calibrated, as further described below, and can be used to estimate how the physical asset interacts with the sample. The prediction results can be used to adjust the operating parameters of the physical asset, and the actual (e.g., experimental) results of the operation can be used to refine and adjust the sample digital twin. This allows the sample digital twin to reconstruct the sample with greater accuracy.
[0048] Digital twins of samples can incorporate diverse information and modeling methods. For example, material properties such as stiffness, stress-strain relationships, electrical conductivity, and resistivity can be included in the digital twin of the sample. In some embodiments, these material properties can be provided by an operator or user to a computer running the digital twin, while in other embodiments, they can be measured from physical assets or generated by computational models (e.g., machine learning models, such as neural network models) based on measurement information. For example, in some embodiments, techniques such as energy-dispersive spectroscopy (EDS) can be used to pre-scan the sample to identify the chemical elements included in the sample, and this chemical element profile can be provided to a machine learning model trained to infer material properties (such as those listed above) from chemical element profiles of certain categories of materials.
[0049] In at least one example, such as for a dual-beam microscope system, a digital twin system can be used to provide increasingly accurate predictions of milling results. For instance, the physical asset digital twin of the digital twin system can initially acquire relevant operating parameters and values to simulate performing an operation using the physical asset. The operation can be simulated to produce the results requested by the operator. The sample digital twin of the digital twin system can be initially calibrated based on sample information provided by the operator and / or obtained using one or more analytical techniques. Then, at least the physical asset digital twin and the sample digital twin can be used in combination to predict the operating parameters to be used. When the operator selects the predicted operating parameters, the actual results can be compared with the requested results, and this comparison can be used to at least update the sample digital twin. In some cases, this comparison can be used to update and adjust the parameters of one or more of the sample digital twin and the physical asset digital twin.
[0050] In some cases, due to the broader range of diversity and variability among samples compared to physical assets, a closed feedback loop between one or more digital twins and physical assets can be used to update at least the sample digital twin (e.g., prioritize the updating of the sample digital twin). For example, physical assets may be associated with relatively stable and easily predictable properties, as the variability of their operation and behavior can be constrained to a known range. In contrast, the various sample types that can be analyzed by physical assets can have a wide range of physical properties, physical states, and / or chemical properties, introducing a certain amount of variability and unpredictability that may hinder accurate simulation. Continuous and prioritized updating of sample digital twins can thereby expand and improve the knowledge base of the digital twin system to allow for more accurate simulation of the samples, and thus more accurate simulation of the interaction between physical assets and samples.
[0051] In at least one embodiment, a sample digital twin can be initially calibrated based on sample-specific information, including but not limited to sample composition, crystal structure, and / or geometry / shape. A high-level overview of the calibration of a digital twin system (e.g., calibration of a sample digital twin of a digital twin system) is provided in [the document / concept]. Figure 10 The following describes and is further illustrated below. In at least one embodiment, where the sample will be analyzed by a charged particle microscopy system (e.g., as a physical asset), initial sample information can be obtained prior to calibration and refinement via analytical techniques such as energy-dispersive X-ray spectroscopy (EDS) and electron energy loss spectroscopy (EELS). For example, data can be obtained via EDS to generate an elemental map of the sample, and data can be obtained via EELS to identify the elemental composition and chemical state of the sample, the electronic structure of the sample material, the thickness of the sample, and / or the dielectric properties of the sample. The data can indicate the shape of the sample interfaces, whether the sample is crystalline or amorphous, and the distances between atoms. Initial sample parameter values can be determined based on the initial sample information.
[0052] In at least one embodiment, the sample digital twin may utilize one or more neural networks, such as a large language model (LLM). The LLM can be trained to facilitate communication between the digital twin system and the controller of the charged particle microscope system, thereby controlling the operation of the charged particle microscope system at least partially based on inferences from the LLM output.
[0053] To calibrate a sample digital twin (which can be used in conjunction with one or more digital twins simulating the interaction between a charged particle microscope system and / or a charged particle beam and a sample), a set of predicted results is obtained using the digital twin system, where initial sample parameter values are applied to the sample digital twin. The digital twin system simulates the operation of the charged particle microscope system to analyze the sample, which produces the predicted results. A set of corresponding experimental results (e.g., actual results obtained through the operation of the charged particle microscope system) is generated by applying the same initial sample parameter values to the system. The sample digital twin can then be calibrated by comparing the set of predicted results with the set of experimental results. For example, the sample parameter values of the sample digital twin can be adjusted based on one or more differences between the set of predicted results and the set of experimental results, thereby allowing for corrective modifications to the sample digital twin.
[0054] The digital twin system (including a calibrated sample digital twin) is then used to plan experiments on a charged particle beam microscope. For example, optimized settings for the charged particle beam microscope can be determined based on sample parameter values from the calibrated sample digital twin. As an example, optimal settings may include values or parameters used to adjust contrast for high-quality images, mitigate distributed beam damage, enhance defect detection capabilities, and / or improve metrological accuracy. These optimal settings can be provided to the operator, for example, at the user interface or graphical user interface (GUI) of the charged particle microscope system controller.
[0055] In at least one implementation, sample digital twins can be calibrated for various sample types to generate a set of calibrated sample digital twins, each calibrated sample digital twin representing a sample type. In some cases, the calibrated sample digital twins do not replicate the physical sample, but instead provide appropriate physical-based predictions of the parameters to be used based on the characteristics of the sample type (e.g., the calibrated sample digital twins do not need to be exact copies of a particular sample).
[0056] Furthermore, when the digital twin of a sample utilizes one or more neural networks (such as LLM), in addition to facilitating the use of a suitable digital twin and optimized instrument settings, LLM can also be used to detect operator-initiated deviations from the optimized instrument settings provided by the digital twin system. For example, during the operation of a charged particle microscope system, the operator may adjust the instrument settings based on one or more of real-time observations and / or technical expertise. This adjustment can be detected by LLM and used as feedback to update the digital twin of the sample. Additionally, experimental results obtained by subsequently applying optimized parameters inferred from the digital twin system and based on a calibrated digital twin of the sample can be used as feedback to update the digital twin of the sample and further improve its accuracy, such as... Figures 10 to 11 As shown. Example
[0057] In one implementation, a digital twin system comprising one or more digital twins configured to simulate physical assets, samples, and / or interactions between physical assets and samples can be used to provide predictions based on information input to the digital twin system. For example, when a physical asset acquires one or more images of a sample, the one or more images, along with sample information, can be input to a computing device that implements the one or more digital twins. Figures 4A to 4B The example illustrates the generation of simulation results based on the input image. Specifically, Figure 4A An image of sample 400 is shown, generated by electron microscopy, and... Figure 4B A digital twin output 450 corresponding to the sample 400 depicted in the image is shown. The image depicts the sample 400 with square grooves 402 of varying depths (e.g., defined along axis 404), which have been milled using a focused ion beam (FIB).
[0058] Specifically, the digital twin output 450 is exemplified as Figure 4B The graph in the image shows the curve along... Figure 4A The depth of sample 400 is intercepted by the dashed line 406. For example, the depth of sample 400 is simulated by a digital twin across the first groove 402a, the second groove 402b, and the third groove 402c, along the direction indicated by arrow 408. Figure 4B In the digital output 450, the graph compares the experimentally measured heights of the first groove 402a, the second groove 402b, and the third groove 402c determined from experimental data 452 (shown as a series of X's). The x-axis of the graph is parallel to the x-axis along the curve. Figure 4AArrow 408 indicates a distance (in μm) across sample 400, and the y-axis of the graph corresponds to depth (e.g., height) (in μm). The depth of sample 400 determined from experimental data 452 is superimposed with synthetic data 454 (e.g., a solid line) predicted by the digital twin of the sample. For example, the graph provides a comparison of experimental depth data and synthetic depth data for the first groove 402a, the second groove 402b, and the third groove 402c. As depicted in the graph, synthetic data 454 produces a predicted depth of groove 402 that is similar to the actual depth measured using experimental data 452.
[0059] Comparison between experimental measurements and predicted measurements can be performed in computational systems (such as...) Figure 1 The digital twin is used in a computing system (100) to continuously update the system's digital twin. For example, such comparisons against a set of representative reference samples with known milling depths (or some other known measurements depending on the type of physical asset) can be used to initialize the accuracy of the digital twin. As an example, the error relative to the known milling depth associated with the predicted measurements generated by the digital twin can be used to correct and optimize the digital twin. Different reference samples can be used to periodically update the digital twin in a similar manner to improve its simulation capabilities, thereby assessing any drift or deviation in the accuracy of the digital twin.
[0060] By utilizing and updating the digital twin of the computational system as described above, the operating parameters of the physical asset can be optimized without relying on sacrificed samples. This allows the desired result to be obtained without the need for trial-and-error testing or the expertise of highly skilled technicians to identify the appropriate parameters. In other words, the closed-loop configuration of the computational system (e.g., between the digital twin system and the physical asset) allows for the automated simulation of the physical asset's operation to achieve the desired result. This can be achieved with reduced user input compared to systems without such closed-loop feedback.
[0061] Furthermore, digital twin systems can utilize one or more physical and chemical models, based on the atomic structure and / or chemical composition of the sample, to predict the effects of operating parameters. Digital twin systems can thus generate simulations with high accuracy because they can account for variations in the sample's material properties during interactions, such as between a charged particle beam and the sample. Therefore, digital twin systems can provide accurate predictions about which instrument setup (e.g., the instrument setup of the physical asset) should be used to achieve the desired effects that facilitate the interaction between the physical asset and the sample. Thus, a digital twin of a computational system can be used both to predict the optimal instrument setup for obtaining target results (e.g., milling depth and / or sample geometry) and to predict results based on specific instrument setups.
[0062] The systems and methods disclosed herein can be used in any of many different applications, including autonomous FIB milling. In autonomous FIB milling, a human user can input a specific desired result (e.g., the shape the sample should be machined into), and a computational system (e.g., ...) Figure 1 The computational system 100 can use a digital twin system (which simulates the sample, FIB pillars, and the interaction between the sample and the FIB pillars in a beam) to output a set of FIB parameters (e.g., beam energy, beam shape, etc.) and patterning strategies (e.g., end milling, vertical milling, etc.) to achieve the desired results. In some embodiments, as described above, modeling the interaction between the beam and the sample digital twin can include a physical model (e.g., one that predicts how the sample will physically respond to being bombarded by an electron beam of a given energy) and a chemical model (e.g., one that predicts the chemical interaction between the sample and the beam based on atomic-level mechanics). The output of the computational system may depend on the parameters of the underlying digital twin (e.g., the chemical composition and material properties of the sample).
[0063] In at least one implementation scheme, Figures 5A to 5D This paper depicts various stages in an exemplary digital twin output of a simulated milling plan for an autonomous FIB milling system. The milling plan guides the removal of material from the edges of a block sample using the FIB beam. The milling plan can be input into a digital twin of the simulated FIB milling system and a digital twin of the simulated sample 500, wherein the milling plan specifies the instrument settings to be used for milling the sample 500. One or more physical and chemical models can be included in the digital twin to allow the digital twin output to predict the operation of the FIB milling system according to the milling plan. The digital twin system may also include a digital twin to simulate the interaction of the FIB beam on the sample using information about the FIB milling system and the sample available from other digital twins.
[0064] like Figure 5A As shown, sample 500 was originally a raw block of material (e.g., not in contact with the FIB milling system). Figure 5B The diagram illustrates an early stage of simulated milling, where a milled region 502 is formed in sample 500 due to the interaction between the FIB milling system and sample 500, causing a portion of sample 500 to be removed along its edge. Figure 5B and Figure 5C As shown, as the simulated milling progresses, the FIB continues to remove material from sample 500, thereby increasing the size of the milled area 502. Figure 5D The final target removal of material from sample 500 was achieved.
[0065] like Figure 5DAs shown, the final geometry (e.g., depth, width, shape, etc.) of the milled region 502 can be predicted based on physical and chemical models implemented using digital twins. For example, different samples may be formed from materials with greater or less resistance to milling removal, or may have chemical compositions that interact differently with the FIB. Therefore, for Figures 5A to 5D For a given milling duration illustrated, different sample materials may have more or less material removed, resulting in milled regions with different shapes as shown. The digital twin system can thus simulate FIB milling of the sample in an automated and accurate manner based on physical and / or chemical models, according to the feedback loop between the physical assets and the digital twin system, for example, as... Figure 1 What is depicted.
[0066] As described above, one or more digital twins can be configured to predict optimized parameters to be applied to a physical asset to perform an operation. As an example, the physical asset could be a dual-beam FIB-SEM system, and the digital twin system could be used as a milling simulator to obtain predicted FIB settings for milling patterns or recipes to achieve the desired results. By implementing a milling simulator, a more accurate estimate of the milling depth resulting from the application of milling patterns (e.g., milling recipes) avoids the trial-and-error determination of optimal milling parameters. For example, without a milling simulator implemented by a digital twin system as described herein, an operator might sacrifice numerous samples used as test specimens to determine a suitable milling pattern. This could stem from using a patterning engine (e.g., software or code for setting milling patterns) provided at the controller of the FIB-SEM system, where the patterning engine utilizes a simplified model to estimate milling parameters (e.g., milling depth, beam overflow, cut face angle, etc.). However, the simplified model may not account for parameter variations and may rely on idealized modeling of the FIB beams. In contrast, the milling simulator uses information received from the FIB-SEM system to compute a rapid simulation of surface evolution, allowing the milling simulator to return more accurate patterned predictions.
[0067] For example, such as Figure 6 As shown, the digital twin system 600 includes a milling simulator communicatively coupled to a controller 602, wherein the milling simulator 604 is implemented by one or more digital twins. The one or more digital twins include at least a digital twin simulating the interaction between the FIB bundle and the sample of a FIB-SEM system (e.g., a physical asset). The milling simulator 604 may also include a digital twin simulating the operation of the FIB-SEM system and / or a digital twin simulating the behavior of the sample. The controller 602 includes one or more processors and other circuitry to control the operation of the FIB-SEM system. Furthermore, the controller 602 is communicatively coupled to various data sources 606, such as databases, data repositories, and storage locations associated with various data collection software.
[0068] The digital twin system 600 utilizes bidirectional information exchange between the controller 602 and the milling simulator 604 to provide recommendations to the operator based on requests input into the digital twin system 600. For example, the operator may request an estimated depth of cut based on a selected milling pattern and a selected time amount, or may request an estimate of parameters of the milling pattern based on the desired depth of cut and / or milling duration.
[0069] In order to generate results based on requests, the milling simulator 604 collects relevant information from the controller 602, as indicated by arrow 608, wherein the controller 602 may collect at least a portion of the relevant information from the data source 606. For example, the milling simulator 604 may obtain from the controller 602 beam properties (such as ion species, ion energy, beam current and / or beam diameter), pattern properties based on x, y and dwell values (such as scan points, scan strategy, number of passes, defocus, etc.), and target properties (such as stage tilt, material and / or application file).
[0070] The milling simulator 604 uses information obtained from the controller 602 to estimate one or more values based on an input request. For example, the milling simulator 604 can be used to obtain estimates of milling depth, number of passes, dose, and / or time. In at least one embodiment, one or more values are estimated based on values from the operator's last or most recent input. For cases where cross-sectional sample (CCS) or rotary cross-section (RCS) milling is to be performed, the milling simulator 604 may further return the pattern space offset relative to the desired edge to be milled and / or the predicted cut face angle.
[0071] One or more values can be predicted based on simulations performed by the milling simulator 604. In one example, in Figure 6 At the first simulation step 612, the milling simulator defines a simulation volume (e.g., a defined three-dimensional space for simulation) and generates simulation objects (e.g., elements that interact within the simulation volume). At the second simulation step 614, as facilitated by digital twins, a rapid surface evolution simulation (e.g., computational modeling of changes in shape and surface over time) is performed on the digital reconstruction of the sample until a stopping condition is met.
[0072] One or more predicted values output by the milling simulator 604 are provided to the controller 602, as indicated by arrow 610. The controller 602 may, based on an operator request, use one or more predicted values to perform one or more operations with the FIB-SEM system to acquire experimental results. The experimental results can be provided from the controller 602 to the milling simulator 604, allowing the milling simulator 604 to compare the experimental results with simulations. For example, the difference between one or more predicted values and the corresponding values obtained from the experimental results can be calculated and used to update one or more digital twins implementing the milling simulator 604. Specifically, the experimental results can be used to update the digital twins configured to simulate samples, thereby improving the accuracy of the digital twins and / or expanding the knowledge base from which the digital twins derive relevant information.
[0073] In another example, artificial intelligence (AI) can be used to implement a digital twin system that leverages one or more digital twins to simulate one or more physical assets, samples, and / or interactions between physical assets and samples. For instance, machine learning algorithms that define the operations of one or more neural networks can be incorporated into an AI-based workflow that relies on a digital twin system to achieve fast and accurate predictions. Using a digital twin system to obtain AI-based predictions can be applied to any use case scenario described herein.
[0074] In at least one implementation, machine learning algorithms can be used to provide a fully automated workflow that reduces the minimum expertise required to operate physical assets efficiently and effectively, while adapting to changes in actual operating conditions. Suboptimal conditions and parameters, as well as deviations from the expected process, can be detected more reliably and robustly than relying solely on operator observation, thus reducing the burden on operators to determine when experimental conditions are not optimal. The automated workflow includes obtaining predictions of optimal settings and conditions for the physical asset, and determining adjustments to the operation of the physical asset in real time during workflow execution.
[0075] However, the effectiveness and robustness of AI-based workflows, and the automation facilitated by them, can depend on how well the AI model (e.g., a neural network) can be trained. For example, the ability of an AI model to accurately predict the optimal instrument setup and conditions for a desired outcome can be constrained by the dataset used to train the AI model. In some cases, such as when physical assets can be analyzed and processed with a wide variety of sample types and experimental conditions, the AI model may not be able to generate accurate predictions for a broad range of samples and scenarios because obtaining a sufficiently large training dataset can be challenging. In at least one implementation, by leveraging a digital twin system, large amounts of diverse training data can be readily obtained, allowing the AI model to generate accurate predictions in real time, despite highly variable experimental environments.
[0076] As an example, Figure 7 The document describes an AI-based workflow 700 for the operation of a FIB system (which may include, for example, a FIB-SEM system), but the AI-based workflow 700 can be applied in a similar manner to other devices and instrumentation systems. Figure 7 As illustrated, the AI-based workflow 700 includes a training and validation unit 702 communicatively connected to the execution unit 704. The training and validation unit 702 is used to generate one or more neural networks (e.g., neural network models) that have been trained and validated based on simulation experiments. In at least one embodiment, the one or more neural networks are trained by a digital twin system. In another embodiment, the one or more neural networks may be trained by a digital twin system or using other algorithms and software to operate as a digital twin system simulating the FIB system to guide the operation of the FIB system in real time.
[0077] For example, a library or collection of simulated milling processes can be used to train a first neural network (e.g., NN1) to predict the outcome of FIB milling, wherein the simulated milling process is generated by one or more digital twins of a digital twin system configured to reconstruct one or more of the FIB system, the sample, and the interaction between the FIB bundle and the sample. In at least one embodiment, it may be as described above and as... Figure 10 The calibration sample digital twin is shown. One or more digital twins can also be deployed using AI-based algorithms (e.g., machine learning), or implemented using other types of non-AI models. The accuracy of one or more digital twins can be verified based on comparisons with experimental results (e.g., as referenced above). Figure 4B and Figure 6 (As described), and a verified digital twin can be deployed to simulate the FIB milling process based on a set of experimental and sample information. Therefore, a large and diverse training dataset can be generated to efficiently train the first neural network.
[0078] The first neural network can be trained to simulate the milling process and milling results based on inputs 706 provided to it (such as different bundle profiles, patterning, scanning strategies, and / or sample types). Upon receiving input 706, the first neural network can construct a simulation environment that mimics the sample environment corresponding to the FIB-SEM system and use input 706 to predict the result 708. Furthermore, the first neural network can learn the correlation between input 706 and milling result 708 (e.g., which sample information, instrument settings, and conditions, etc.) based on a training dataset generated by a digital twin system.
[0079] For example, result 708 may include the temporal evolution of the milled surface and the generation of auxiliary signals, such as, but not limited to, stage current measurement and / or secondary charged particles. During training, the first neural network may use simulations provided by the digital twin system to determine the correlation between the detection of stage current and / or secondary charged particles and the milling depth and / or surface topography. In at least one embodiment, training may be performed offline, for example, not during the operation of the FIB system. Furthermore, training may be performed on a different computing device than one or more computing devices used to control the FIB system and / or host the AI-based workflow 700.
[0080] In at least one implementation, offline training simplifies the workflow for operating the FIB-SEM system by avoiding the need for trial and error to determine optimal settings and milling parameters. For example, without using AI-based workflow 700, the workflow could include performing simulations and / or other time-consuming processes as an initial step to identify the appropriate milling pattern to use, thereby lengthening the workflow and reducing throughput.
[0081] The results 708 inferred by the first neural network based on the simulation milling process can be validated using a selected simulation (e.g., selected from simulations generated by a digital twin system). For example, simulations representing different experimental scenarios can be selected for the validation dataset, where the validation dataset differs from the training dataset (e.g., simulations not used for training are used for validation). Once validated, the trained first neural network can be used to generate milling predictions 710 for the execution part 702 provided to the AI-based workflow 700, as indicated by arrow 712.
[0082] An execution unit 702 is deployed at a computing device hosting an AI-based workflow 700. At the computing device, the execution unit 702 receives operator input 714, such as known sample information and desired results. For example, known sample information may include the composition and distribution of structural and / or constituent components (e.g., homogeneous or heterogeneous), and if heterogeneous, the spatial arrangement of the structural and / or constituent components. In at least one example, the computing device may include a GUI that allows the operator to create a virtual three-dimensional (3D) model of the sample, or to create or input one or more voice commands corresponding to known sample information, a pre-existing virtual 3D model (such as a CAD model), or a textual description of the sample. Desired results may include target values to be obtained via the AI-based workflow 700, such as a target milling depth. Desired results may also include specifications such as the milling location at the sample, the target milling shape, the spatial dimensions of the features milled at the milling location, time constraints, and / or non-compliance tolerances of the result specifications.
[0083] The trained first neural network receives operator input 714 and uses the operator input 714 to output one or more inferences 716. The trained first neural network also uses system state information 718 to generate one or more inferences 716. For example, an AI-based workflow 700 may include software algorithms to provide system state information to the trained first neural network. The system state information includes one or more of the following: the spatial arrangement and location of the FIB source, the voltage applied to the focusing lens and other beamforming devices of the FIB system, and the physical location of any beam modulation apertures of the FIB system.
[0084] One or more inferences 716 generated by a trained first neural network based on operator input 714 and system state information 718 are provided to the FIB system controller, enabling the FIB system controller to apply the one or more inferences 716 to a milling operation to be performed by the FIB system. For example, the one or more inferences 716 include predictive initial settings and parameters for the milling operation to obtain the desired result. The milling operation is performed using the one or more inferences, and the trained first neural network performs real-time observation 720 during the duration of the milling operation. Real-time observation 720 includes monitoring the progress of the milling operation and comparing that progress with the corresponding prediction result. For example, the trained first neural network can periodically determine the milling depth within set time intervals during the duration of the milling operation, wherein the milling depth can be measured based on signals generated during milling and / or periodic acquisition of images of the milling site.
[0085] In cases where milling progress is monitored based on images acquired during milling (e.g., electronic images and / or FIB images), a trained second neural network (e.g., Figure 1 Image recognition 722 is performed using a second neural network (NN2). For example, the trained second neural network can be trained and optimized to identify image features corresponding to sample milling, such as surface topography and the formation of milling features during milling. The trained second neural network can be used to evaluate the evolution of the milled surface of the sample and confirm real-time signal feedback based on signals during the process.
[0086] The real-time observation 720 performed by the trained first neural network may also include inferring modifications to the milling process based on real-time signal feedback (e.g., by adjusting milling parameters, FIB system settings, etc.). For example, the trained first neural network may identify deviations from the expected milling progress (e.g., deviations from the expected value exceeding a predetermined amount) and infer one or more adjustments to the milling process to correct the deviations. Thus, the AI-based workflow 700 may include iterative processes between generating one or more inferences 716 and performing real-time observation 720 on the milling process until the milling process is complete. Upon completion, the final milling feature 724 is created.
[0087] In at least one embodiment, the trained first neural network can be implemented as a digital twin system for a FIB system (or FIB-SEM system) and can simulate one or more of the interactions between a sample, the FIB system, and / or the FIB bundle and the sample. For example, although mentioned in the singular, the trained first neural network may include more than one neural network model (e.g., a model network) such that each model can be trained to perform different tasks (such as sample simulation, FIB system simulation, and bundle-sample interaction simulation). Alternatively, different layers of the first neural network can be configured to simulate the sample, physical assets, and interactions between the sample and the physical assets. As described above, the first neural network can be trained using simulations generated by the digital twin system or via simulations generated using some other technology or software. The trained first neural network is then deployed itself as a digital twin system during operation of the FIB system.
[0088] For example, a trained first neural network can simulate a milling process based on operator input 714 and system state information 718, and determine the optimal instrument settings to produce the requested result. As described above, the trained first neural network can monitor the progress of the milling experiment and compare the experimental progress with the simulated milling process to predict and provide real-time adjustments to the instrument settings. When a stopping condition is met (e.g., reaching the requested duration or achieving the requested milling depth), deviations from the expected (e.g., predicted) instrument settings are determined. For example, if the operator requests that the milling process continue until the target depth is reached, the digital twin system can be updated using the deviation from the predicted time amount or the deviation from the predicted milling point shape when the target depth is achieved. As an example, the weights of the first neural network corresponding to the simulation of the sample can be updated based on the deviations. In this way, the digital twin system can be trained and optimized based on physical assets (e.g., based on training data generated by another digital twin system) and can be continuously updated upon deployment. Exemplary systems and operating methods for implementation
[0089] Figure 8 This is a block diagram of a scientific instrument support module 800 for performing support operations according to various embodiments. The scientific instrument support module 800 may be implemented by circuitry (e.g., including electrical and / or optical components), such as a programmed computing device (e.g., Figure 1 Computing device 114 or Figure 14 (Any of the computing devices 1400 shown). The logical components of the scientific instrument support module 800 may be included in a single computing device, or, as appropriate, distributed across multiple computing devices communicating with each other. (References herein) Figure 14The computing device 1400 discusses examples of computing devices that can implement the scientific instrument support module 800 alone or in combination, and this document references Figure 15 The scientific instrument support system 1500 discusses an example of a system of interconnected computing devices, wherein the scientific instrument support module 800 can be implemented across one or more of these computing devices. Furthermore, the scientific instrument support module 800 may include... Figure 2 The component model 200, wherein the component model can be distributed across the various logic elements of the scientific instrument support module 800.
[0090] Scientific instrument support module 800 may include a first logic unit 802, a second logic unit 804, and a third logic unit 806. As used herein, the term "logic unit" may include means for performing a set of operations associated with that logic unit. For example, any logic element included in support module 800 may be implemented by one or more computing devices programmed with instructions to cause one or more processing devices of the computing device to perform an associated set of operations. In a particular embodiment, a logic element may include one or more non-transitory computer-readable media having instructions on them that, when executed by one or more processing devices of the one or more computing devices, cause the one or more computing devices to perform an associated set of operations. As used herein, the term "module" may refer to a collection of one or more logic elements that together perform the functions associated with the module. Different logic elements in a module may take the same form or may take different forms. For example, some logic units in a module may be implemented by a programmed general-purpose processing device, while other logic units in the module may be implemented by an application-specific integrated circuit (ASIC). In another example, different logical elements within a module may be associated with different sets of instructions executed by one or more processing devices. A module may not include all the logical elements depicted in the associated figures; for example, a module may include a subset of the logical elements depicted in the associated figures when it is to perform a subset of the operations discussed herein with reference to that module. Furthermore, in at least one embodiment, the algorithm (e.g., code) applied at the module may include executable instructions for performing AI-based (e.g., one or more neural network) processes and operations. For example, a digital twin of a digital twin system implemented at scientific instrument support module 800 may include one or more neural networks.
[0091] According to any of the embodiments disclosed herein, the first logic component 802 can provide computational simulation of a scientific instrument (e.g., a physical asset). For example, the first logic component 802 can be used in a digital twin that models instrument behavior (e.g., generates predictions) based on instrument specifications. As an example, a computing device that hosts both a scientific instrument support module 800 and a control system for the scientific instrument, or a computing device that implements the control system for the scientific instrument and is communicatively coupled to the computing device hosting the scientific instrument support module 800, can provide instrument specifications to the digital twin. The first logic component 802 can use the instrument specifications to facilitate the simulation of instrument operation according to, for example, a given set of operating parameters (such as a milling schedule).
[0092] According to any of the embodiments disclosed herein, the second logic unit 804 can provide computational simulation of the sample. For example, the second logic unit 804 can be used in a digital twin to model the physical and chemical properties of the sample. In at least one embodiment, the second logic unit 804 can utilize physical and chemical models corresponding to the sample and provided to it by the user to allow the digital model to simulate the behavior of the sample under various conditions, such as analytical environments.
[0093] According to any embodiment disclosed herein, the third logic component 806 can provide a computational simulation of the interaction between the scientific instrument and the sample. For example, the third logic component 806 can be used in a digital twin to model the results of applying the scientific instrument to a sample. The third logic component 806 can combine with the first logic component 802 and the second logic component 804 to generate predictions, for example, by utilizing simulations of the scientific instrument's behavior and simulations of the sample's properties. This allows the digital twin to accurately estimate the effects of operating the scientific instrument's behavior on the sample.
[0094] Figure 9 This is a high-level flowchart of method 900 for supporting operations of scientific instruments (e.g., physical assets) according to various implementation schemes. Method 900 consists of a managed scientific instrument support module (such as... Figure 8 The scientific instrument support module 800) is executed by a computing device (e.g., one or more processors). The computing device includes a computing system (e.g., Figure 1 In the computing system 100, the computing system includes physical assets communicatively coupled to the digital twin system via a closed feedback loop, such that information stored at and used by the control system of the physical assets can be accessed by the digital twin system. Furthermore, the computing device includes a memory where executable instructions for performing method 900 are stored. The computing device may also implement a control system for the physical assets (e.g., scientific instruments) or be communicatively coupled to another computing device that implements the control system. While the operation of method 900 and... Figures 10 to 12The methods described herein and further described below may be referenced to specific embodiments disclosed herein (e.g., references to...). Figure 8 The scientific instrument support module 800 discussed in this article is referenced. Figure 13 The GUI 1300 discussed in this article is referenced. Figure 14 The computing device 1400 discussed and / or referenced herein Figure 15 The scientific instrument support system 1500 discussed is used to illustrate this, but method 900 (and...) Figures 10 to 12 The method can be used in any suitable setup to perform any appropriate support operation. Figure 9 In this method, operations are each exemplified once in a specific order, but operations can be reordered and / or repeated as needed and as appropriate (e.g., different operations can be performed in parallel where appropriate). Furthermore, in at least one embodiment, at least a portion of method 900 may utilize artificial intelligence (e.g., one or more neural networks) to infer the evolution of processes and operations and / or predict the outcomes of processes and operations.
[0095] At 902, method 900 includes receiving instrument and sample information. For example, a user may provide operating parameters and / or instrument specifications of a scientific instrument, for instance, at a user interface of a computing device, for the simulation of instrument behavior and performance facilitated by one or more digital twins. The user may further provide information about the material type, state, and any prior treatment of the sample to be simulated by one or more digital twins.
[0096] At 904, method 900 includes receiving a request for results. For example, a user may input desired simulation results into the digital twin system. As an example, the desired simulation results could be a set of operating parameters for a scientific instrument, predicted to produce a target effect of the interaction between the scientific instrument and the sample. For example, the desired simulation results could be operating parameters for a FIB system milling grooves of a specific shape and size in a sample. As another example, the desired simulation results could be the effect of the interaction between the scientific instrument and the sample on the sample. For example, the desired simulation results could be an image and corresponding metrology that determines how much material is removed from a sample by the FIB using a given set of operating parameters for a specific type of sample.
[0097] At 906, method 900 includes outputting simulation results based on a simulation performed by the digital twin system and upon receiving a request for results. For example, as described above, the results may include operating parameters for the scientific instrument, data regarding the results of interactions between the scientific instrument and the sample, and / or other simulation results associated with the scientific instrument and / or the sample. In at least one embodiment, the results may be displayed to a user at a user interface of a computing device, relayed to another computing device, stored in memory or a database, etc.
[0098] Figure 10 This is a flowchart of a method 1000 for performing support operations to calibrate a digital twin system including one or more digital twins and using the digital twin system to optimize physical assets, according to various implementation schemes. Method 1000 can be executed by a computing device having one or more processors, wherein the computing device may include a computing system having physical assets communicatively coupled to the digital twin system (e.g., ...). Figure 1 In a computing system 100), the digital twin system can be configured to simulate one or more of a physical asset, a sample, and the interaction between the physical asset and the sample. The computing device includes a memory where executable instructions for performing method 1000 are stored, and also implements a control system for the physical asset (e.g., a scientific instrument). Figure 10 In this method, operations are each exemplified once in a specific order, but operations can be reordered and / or repeated as needed and as appropriate (e.g., different operations can be performed in parallel where appropriate). Furthermore, in at least one embodiment, at least a portion of method 1000 may utilize artificial intelligence (e.g., one or more neural networks) to infer the evolution of processes and operations and / or predict the outcomes of processes and operations.
[0099] At 1002, method 1000 includes acquiring one or more initial datasets. In at least one embodiment, the one or more initial datasets may include parametric digital twins modeled on a particular type of sample (e.g., based on composition, crystal structure, geometry) and / or any sample information input by the operator, as referenced above. Figure 7 The operator input 714 describes this. One or more initial datasets may also include information about the physical asset's settings, characteristics, and / or current state. For example, system state information (such as...) can be obtained from the physical asset's controller. Figure 7(System status information 718). Additionally, one or more initial datasets may include data obtained from various databases, repositories, and other storage locations that store experimental and simulation processes and results. Furthermore, one or more initial datasets may also include a set of experiments and measurements used to determine the initial parameters of the digital twin system (e.g., initial values of a sample digital twin configured to simulate a sample). For example, as previously described, data collected using EDS and / or EELS can be used to obtain sample information from which appropriate initial sample parameter values can be identified.
[0100] At 1004, method 1000 includes performing a set of experiments to obtain a corresponding set of experiment-based measurements and results for calibrating the digital twin system. For example, this set of experiments can be performed using physical assets and a sample to be modeled to obtain the set of experiment-based measurements and results, or, if data based on this set of experiments is already available, the set of experiment-based measurements and results can be retrieved from a storage device. Corresponding predicted measurements and / or predicted results are obtained from the digital twin system and compared with the set of experiment-based measurements and results. This comparison can be used to correct deviations between the digital twin system and the expected results determined based on the set of experiment-based measurements and results. For example, initial values for the sample digital twin can be adjusted based on the comparison. As an example, when one or more neural networks are used to implement the sample digital twin, a loss can be calculated based on the comparison (e.g., using a loss function), and this loss is used to update the weights of one or more neural networks to achieve a minimum threshold level of accuracy. In at least one embodiment, further calibration of the digital twin system can be performed until the minimum threshold level of accuracy is achieved.
[0101] At 1006, method 1000 includes using a calibrated digital twin system to optimize the setup of a physical asset. For example, when the physical asset is a FIB-SEM system, an operator can indicate desired results and / or desired operations to be performed. As an example, the operator can enter information about the desired results and / or operations at the GUI of the FIB-SEM controller. The calibrated digital twin system can receive the information and use it to output optimization parameters, such as settings for controlling contrast, distributed bundle damage, defect detection, and / or metrological accuracy.
[0102] At 1008, method 1000 includes applying optimized settings to the physical asset when instructing it to perform a desired operation. For example, applying optimized settings may include, for instance, presenting optimized settings determined by the digital twin system to an operator at a GUI. If confirmation to continue the operation is received from the operator, the optimized settings are used at the physical asset to perform the operation. The application of optimized instrument settings may also be based on… Figure 11 The information identified at point 1106 is used to execute the procedure, as further described below.
[0103] At 1010, method 1000 includes an application based on optimized instrument settings to acquire experimental data from the physical asset. For example, the operation can be performed by the physical asset. Measurements, including metrological data, can be obtained as a result of the operation and stored at a storage location (e.g., in memory, a database, etc.). Additionally, detection of any deviations from the optimized instrument settings initiated by the operator can be recorded. For example, when the physical asset is an electron microscope, the operator can adjust the contrast settings based on visual observation while the operation is in progress. Figure 11 As described, this adjustment can be recorded and used to update the digital twin system.
[0104] Figure 11 This is a flowchart of method 1100 for performing update and / or training operations for a digital twin system according to various implementation schemes. Method 1100 is supported by a managed scientific instrumentation module (such as...) Figure 8 The scientific instrument support module 800) is executed by a computing device (e.g., having one or more processors). The computing device includes a computing system (e.g., Figure 1 In the computing system 1100, the computing system includes one or more physical assets of a digital twin that are communicatively coupled to the digital twin system via a closed feedback loop, such that information stored at and used by the control system of the physical assets can be accessed by the one or more digital twins. Furthermore, the computing device includes a memory where executable instructions for performing method 1100 are stored. The computing device may also be a controller for the physical assets (e.g., scientific instruments), or another computing device communicatively coupled to control the physical assets. In Figure 1100, operations are each illustrated once in a specific order, but operations can be reordered and / or repeated as needed and as appropriate (e.g., different operations can be performed in parallel where appropriate). Furthermore, in at least one embodiment, at least a portion of method 1100 may utilize artificial intelligence (e.g., one or more neural networks) to infer the evolution of processes and operations and / or predict the outcomes of processes and operations.
[0105] At 1102, method 1100 includes comparing experimental data obtained from physical assets with simulated data generated by a digital twin system. For example, as described above... Figure 10 As described, results acquired by physical assets using optimized instrument settings can be compared with simulation results output by a digital twin system based on optimized instrument settings.
[0106] At 1104, method 1100 includes determining the difference between experimental data and simulation results. In one example, such as when implementing a digital twin system using one or more neural networks, this difference is used to calculate a loss via a loss function.
[0107] At 1106, method 1100 includes updating the digital twin system based on the difference. Updating the digital twin system may include modifying the initial or recently used model parameters of the digital twin system. For example, as... Figure 4B As shown, milling of a sample by a FIB can be simulated, and the depth of one or more simulated grooves milled in the sample can be measured. The simulated depth measurement can be compared with the known indentation depth of a reference sample prepared using the same instrument setup and sample type as in the simulation. The comparison can be used to determine the difference or amount of error. In one embodiment, at least a sample digital twin of the digital twin system (e.g., a digital twin modeling the behavior and properties of a type of sample) can be updated to improve the accuracy of the sample digital twin relative to how the sample digital twin can simulate the interaction with physical assets. The sample digital twin can be updated by changing the values corresponding to sample parameters (such as composition, crystal structure, and geometry) used by the sample digital twin. Furthermore, when the sample digital twin includes one or more neural networks, updating the sample digital twin includes calculating gradients based on the loss and adjusting the weights of the neural networks based on the gradients. Method 1100 can then return to Figure 10 1008, to apply the updated settings of the digital twin system to physical assets.
[0108] Figure 12 This is a flowchart of method 1200 for training an AI-based model (e.g., utilizing one or more neural networks), which can be implemented in a digital twin system to simulate one or more of a physical asset, a sample, and the interaction between the physical asset and the sample. Method 1200 is compatible with managed scientific instrument support modules (e.g., Figure 8 The scientific instrument support module 800 is executed by a computing device that is different from or the same as the computing device (e.g., having one or more processors). The computing device includes a memory where executable instructions for executing method 1200 are stored. In Figure 1200, the operations are each illustrated once in a specific order, but the operations can be reordered and / or repeated as needed and as appropriate (e.g., different operations can be executed in parallel where appropriate).
[0109] At 1202, method 1200 includes obtaining one or more datasets relevant to training one or more neural networks. For example, one or more datasets may be retrieved from one or more data sources. In some cases, the one or more datasets may be preprocessed into a suitable format for training one or more neural networks. The one or more datasets may be further split into subsets for specific operations. As an example, the one or more datasets may be split at least into a training subset for fitting an AI-based model and a validation subset for tuning or refining the model.
[0110] At 1204, method 1200 includes initializing an AI-based model. In at least one embodiment, initializing the model includes defining the architecture of one or more neural networks. For example, the number of layers, the type of layers, and the number of neurons per layer may be specified. Initializing the model may also include determining the initial weights and biases of one or more neural networks.
[0111] At 1206, method 1200 includes performing a forward propagation. For example, input data from a training subset can be passed through one or more neural networks to generate predictions, which may include sequentially computing the output of each layer of the neural network—that is, from the input layer to the output layer of the neural network.
[0112] At 1208, method 1200 includes calculating a loss. The loss can be the difference between the predicted output of one or more neural networks and the expected outcome (e.g., the true value). For example, the expected outcome may correspond to labeled data included in a subset of the training set. The loss can be calculated using any one or more loss functions, such as mean squared error (MSE), cross-entropy, or some other type of loss function.
[0113] At 1210, method 1200 includes performing backpropagation to determine the error through the backpropagation of one or more neural networks. Gradients are computed relative to the weights and biases of the neural networks.
[0114] At 1212, method 1200 includes updating the weights and biases of one or more neural networks. The weights and biases are updated based on the computed gradients using optimization algorithms such as stochastic gradient descent (SGD) or adaptive moment estimation (Adam).
[0115] At 1214, method 1200 includes using a validation subset to validate the AI-based model. Validation of the AI-based model allows for monitoring of the model's performance during training. For example, validation results can be used to assess how well the model generalizes to data not used to train the model, thereby mitigating overfitting.
[0116] At 1216, method 1200 includes confirming convergence based on the performance of the AI-based model. In at least one implementation, convergence is determined based on the loss decreasing below a predetermined threshold, or the model's performance on a validated subset reaching a minimum threshold level. If convergence is not achieved, method 1200 includes returning to 1206 to perform a forward propagation for further training. If convergence is achieved, method 1200 includes advancing to 1218 to use the AI-based model. For example, the trained model can be applied to test data to evaluate the model's final performance on unseen data (e.g., new data). Additionally or alternatively, the trained model can be deployed at a computing device hosting the digital twin system.
[0117] The scientific instrument support methods disclosed herein may include (e.g., via references herein) Figure 15 The user-local computing device (1520) discussed here interacts with human users. These interactions may include providing information to the user (e.g., about scientific instruments such as...) Figure 15 Information on the operation of the scientific instrument (1510), information about the sample being analyzed or other tests or measurements performed by the scientific instrument, information retrieved from local or remote databases or other information, or information provided to the user for inputting commands (e.g., for controlling the scientific instrument (such as...) Figure 15 The scientific instrument 1510 provides options for operating the instrument, or for controlling the analysis of data generated by the scientific instrument, querying (e.g., querying a local or remote database), or other information. In some embodiments, these interactions can be performed via a graphical user interface (GUI), which includes a display device (e.g., referenced herein). Figure 14 The display device 1410 discussed herein is a visual display that provides output and / or prompts to the user (e.g., via reference herein). Figure 14 Input may be provided by one or more input devices included in the other I / O devices 1412 discussed herein, such as a keyboard, mouse, trackpad, or touchscreen. The scientific instrument support system disclosed herein may include any suitable GUI for user interaction.
[0118] Figure 13 Example GUI 1300, according to various embodiments, can be used to perform some or all of the support methods disclosed herein. As described above, GUI 1300 can be set up in a scientific instrument support system (e.g., as referenced herein). Figure 15 The computing device of the scientific instrument support system 1500 discussed herein (e.g., referenced herein) Figure 14 The display device of the computing device 1400 discussed herein (e.g., referenced herein) Figure 14The display device 1410 discussed herein can be used, and the user can use any suitable input device (e.g., the one referred to herein). Figure 14 The other I / O devices 1412 discussed herein include any input devices and input technologies (e.g., cursor movement, motion capture, facial recognition, gesture detection, speech recognition, button actuation, etc.) that interact with the GUI 1300.
[0119] The GUI 1300 may include a data display area 1302, a data analysis area 1304, a scientific instrument control area 1306, and a settings area 1308. Figure 13 The specific number and arrangement of areas depicted are merely illustrative, and the GUI may include any number and arrangement of areas (including any desired features).
[0120] Data display area 1302 can display physical assets (such as scientific instruments, for example, as referenced in this article). Figure 15 The data generated by the scientific instrument 1510 under discussion. For example, data display area 1302 can display a comparison between measured values of physical assets in the digital twin system and simulations of the same values (e.g., such as...). Figure 4B (As shown).
[0121] Data analysis area 1304 can display the results of data analysis (e.g., the results of analyzing the data illustrated in data display area 1302 and / or other data). For example, data analysis area 1304 can display a recommended set of FIB parameters and patterning strategies for autonomous FIB milling, as discussed above. In some embodiments, data display area 1302 and data analysis area 1304 can be combined in GUI 1300 (e.g., to include data output from scientific instruments and some analysis of the data in a public graphic or area).
[0122] Scientific instrument control area 1306 may include a function that allows users to control scientific instruments (e.g., as referenced herein). Figure 15 Options for the scientific instrument 1510 under discussion. For example, the scientific instrument control area 1306 may include the current operating parameters of the scientific instrument (which may be automatically set by feedback between the scientific instrument and the digital twin system, or manually set by a human technician).
[0123] Setting up area 1308 may include features and functions that allow users to control GUI 1300 (and / or other GUIs) and / or perform common computational operations on data display area 1302 and data analysis area 1304 (e.g., storing data on storage devices such as those referenced herein). Figure 14 Options for (such as) transferring data to another user, marking data, etc. on the storage device 1404 under discussion.
[0124] As mentioned above, Figure 8 The scientific instrument support module 800 can be implemented by one or more computing devices. Figure 14 This is a block diagram of a computing device 1400 capable of performing some or all of the scientific instrument support methods disclosed herein, according to various embodiments. In some embodiments, the scientific instrument support module 800 may be implemented by a single computing device 1400 or by multiple computing devices 1400. Furthermore, as discussed below, the computing device 1400 (or multiple computing devices 1400) implementing the scientific instrument support module 800 may be... Figure 15 A part of one or more of the scientific instrument 1510, the user local computing device 1520, the service local computing device 1530, or the remote computing device 1540.
[0125] Figure 14 The computing device 1400 is illustrated as having multiple components, but any one or more of these components may be omitted or repeated depending on their suitability for the application and setup. In some embodiments, some or all of the components included in the computing device 1400 may be attached to one or more motherboards and encapsulated in a housing (e.g., including plastic, metal, and / or other materials). In some embodiments, some of these components may be fabricated onto a single system-on-a-chip (SoC) (e.g., the SoC may include one or more processing devices 1402 and one or more storage devices 1404). Additionally, in various embodiments, the computing device 1400 may not include... Figure 14 The device may include one or more of the components illustrated herein, but may also include interface circuitry (not shown) for coupling to one or more components using any suitable interface, such as a Universal Serial Bus (USB) interface, a High Definition Multimedia Interface (HDMI) interface, a Controller Area Network (CAN) interface, a Serial Peripheral Interface (SPI) interface, an Ethernet interface, a wireless interface, or any other suitable interface. For example, computing device 1400 may not include display device 1410, but may include display device interface circuitry (e.g., connector and driver circuitry) to which display device 1410 may be coupled.
[0126] Computing device 1400 may include processing device 1402 (e.g., one or more processing devices). As used herein, the term "processing device" may refer to any device or part of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory. Processing device 1402 may include one or more digital signal processors (DSPs), application-specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), cryptographic processors (dedicated processors that execute cryptographic algorithms within hardware), server processors, or any other suitable processing device.
[0127] Computing device 1400 may include storage device 1404 (e.g., one or more storage devices). Storage device 1404 may include one or more memory devices, such as random access memory (RAM) (e.g., static RAM (SRAM) devices, magnetic RAM (MRAM) devices, dynamic RAM (DRAM) devices, resistive RAM (RRAM) devices, or conductive bridged RAM (CBRAM) devices), hard disk drive-based memory devices, solid-state memory devices, network drives, cloud drives, or any combination of memory devices. In some embodiments, storage device 1404 may include memory sharing a die with processing device 1402. In such embodiments, the memory may be used as cache memory and may include, for example, embedded dynamic random access memory (eDRAM) or spin-transfer torque magnetic random access memory (STT-MRAM). In some embodiments, storage device 1404 may include a non-transitory computer-readable medium having instructions on which, when executed by one or more processing devices (e.g., processing device 1402), cause computing device 1400 to perform any suitable method or portion thereof of the methods disclosed herein.
[0128] Computing device 1400 may include interface device 1406 (e.g., one or more interface devices 1406). Interface device 1406 may include one or more communication chips, connectors, and / or other hardware and software to manage communication between computing device 1400 and other computing devices. For example, interface device 1406 may include circuitry for managing wireless communication used to transmit data to and from computing device 1400. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can transmit data through a non-solid medium using modulated electromagnetic radiation. This term does not imply that the associated device does not contain any wires, although in some embodiments it may not contain any wires. The circuitry included in interface device 1406 for managing wireless communications can implement any of a number of wireless standards or protocols, including but not limited to Institute of Electrical and Electronics Engineers (IEEE) standards, including Wi-Fi (IEEE 802.11 series), IEEE 802.16 standards (e.g., IEEE 802.16-2005 amendments), Long Term Evolution (LTE) projects, and any amendments, updates, and / or revisions (e.g., Advanced LTE projects, Ultra Mobile Broadband (UMB) projects (also known as “3GPP2”), etc.). In some implementations, the circuitry included in interface device 1406 for managing wireless communications can operate according to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE networks. In some embodiments, the circuitry included in interface device 1406 for managing wireless communications may operate according to Enhanced Data GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). In some embodiments, the circuitry included in interface device 1206 for managing wireless communications may operate according to Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolved Data Optimization (EV-DO), and their derivative protocols, as well as any other wireless protocols designated as 3G, 4G, 5G, and higher versions. In some embodiments, interface device 1406 may include one or more antennas (e.g., one or more antenna arrays) for receiving and / or transmitting wireless communications.
[0129] In some embodiments, interface device 1406 may include circuitry for managing wired communications, such as electrical communication protocols, optical communication protocols, or any other suitable communication protocol. For example, interface device 1406 may include circuitry for supporting communications according to Ethernet technology. In some embodiments, interface device 1406 may support both wireless and wired communications, and / or may support multiple wired communication protocols and / or multiple wireless communication protocols. For example, a first set of circuitry for interface device 1406 may be dedicated to shorter-range wireless communications, such as Wi-Fi or Bluetooth, and a second set of circuitry for interface device 1406 may be dedicated to longer-range wireless communications, such as Global Positioning System (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV-DO, etc. In some embodiments, a first set of circuitry for interface device 1406 may be dedicated to wireless communications, and a second set of circuitry for interface device 1406 may be dedicated to wired communications.
[0130] The computing device 1400 may include a battery / power circuit 1408. The battery / power circuit 1408 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 1400 to a power source (e.g., AC line power) that is separate from the computing device 1400.
[0131] The computing device 1400 may include a display device 1410 (e.g., multiple display devices). The display device 1410 may include any visual indicator, such as a head-up display, computer monitor, projector, touch screen display, liquid crystal display (LCD), light-emitting diode display, or flat panel display.
[0132] The computing device 1400 may include other input / output (I / O) devices 1412. Other I / O devices 1412 may include, for example, one or more audio output devices (e.g., speakers, headphones, earphones, alarms, etc.), one or more audio input devices (e.g., microphones or microphone arrays), positioning devices (e.g., GPS devices that communicate with satellite-based systems to receive the location of the computing device 1400, as known in the art), audio codecs, video codecs, printers, sensors (e.g., thermocouples or other temperature sensors, humidity sensors, pressure sensors, vibration sensors, accelerometers, gyroscopes, etc.), image capture devices (such as cameras), keyboards, cursor control devices (such as mice, styluses, trackballs, or touchpads), barcode readers, quick-response (QR) code readers, or radio frequency identification (RFID) readers.
[0133] The computing device 1400 may have any suitable form factor for its application and setup, such as a handheld or mobile computing device (e.g., a cellular phone, smartphone, mobile internet device, tablet computer, laptop computer, netbook computer, ultrabook computer, personal digital assistant (PDA), ultra-mobile personal computer, etc.), a desktop computing device or server computing device or other networked computing component.
[0134] One or more computing devices that implement any of the scientific instrument support modules or methods disclosed herein may be part of a scientific instrument support system. Figure 15 This is a block diagram of an example scientific instrument support system 1500, according to various implementation schemes, in which some or all of the scientific instrument support methods disclosed herein can be performed. The scientific instrument support modules and methods disclosed herein (e.g., Figure 8 Scientific Instrument Support Module 800 and Figures 9 to 12 Methods 800 to 1200 can be implemented by one or more of the scientific instrument 1510, user local computing device 1520, service local computing device 1530, or remote computing device 1540 of the scientific instrument support system 1500.
[0135] Any of the scientific instrument 1510, the user local computing device 1520, the service local computing device 1530, or the remote computing device 1540 may be included in the references herein. Figure 14 Any implementation of the computing device 1400 discussed herein, and any of the scientific instrument 1510, user local computing device 1520, service local computing device 1530, or remote computing device 1540 may be adopted as referenced herein. Figure 14 The form of any suitable implementation of the computing device 1400 under discussion.
[0136] Scientific instrument 1510, user local computing device 1520, service local computing device 1530, or remote computing device 1540 may each include processing device 1502, storage device 1504, and interface device 1506. Processing device 1502 may take any suitable form, including those referenced herein. Figure 14 The processing device 1502 discussed herein may take any form, and the processing device 1502 included in different devices such as scientific instrument 1510, user local computing device 1520, service local computing device 1530, or remote computing device 1540 may take the same or different forms. Storage device 1504 may take any suitable form, including those referenced herein. Figure 14The storage device 1504 discussed herein may take any form, and the storage device 1504 included in different devices such as scientific instrument 1510, user local computing device 1520, service local computing device 1530, or remote computing device 1540 may take the same or different forms. Interface device 1506 may take any suitable form, including those referenced herein. Figure 14 The interface device 1406 discussed may take any form, and the interface device 1506 included in different devices such as scientific instrument 1510, user local computing device 1520, service local computing device 1530, or remote computing device 1540 may take the same or different forms.
[0137] Scientific instrument 1510, user local computing device 1520, service local computing device 1530, and remote computing device 1540 can communicate with other components of scientific instrument support system 1500 via communication path 1508. Communication path 1508 can communicatively couple interface devices 1506 (as shown) of different components within scientific instrument support system 1500, and can be a wired or wireless communication path (e.g., according to references herein). Figure 14 The interface device 1406 of the computing device 1400 is any of the communication technologies discussed in the communication technology. Figure 15 The specific scientific instrument support system 1500 depicted includes communication paths between each pair of devices among scientific instrument 1510, user local computing device 1520, service local computing device 1530, and remote computing device 1540. However, this specific implementation of "full connectivity" is merely exemplary, and various communication paths in communication path 1508 may not exist in various embodiments. For example, in some embodiments, service local computing device 1530 may not have a direct communication path 1508 between its interface device 1506 and the interface device 1506 of scientific instrument 1510, but may communicate with scientific instrument 1510 via communication path 1508 between service local computing device 1530 and user local computing device 1520 and communication path 1508 between user local computing device 1520 and scientific instrument 1510.
[0138] Scientific instrument 1510 may include any suitable scientific instrument, such as charged particle microscope (e.g., electron microscope, such as TEM or SEM), FIB system or charged particle dual-beam microscope / system.
[0139] User-local computing device 1520 may be a user-local computing device of scientific instrument 1510 (e.g., any embodiment of the computing device 1400 discussed herein). In some embodiments, user-local computing device 1520 may also be located locally to scientific instrument 1510, but this is not mandatory; for example, user-local computing device 1520 in a user's home or office may be located remotely to scientific instrument 1510, but communicate with the scientific instrument so that the user can use user-local computing device 1520 to control and / or access data from scientific instrument 1510. In some embodiments, user-local computing device 1520 may be a laptop computer, smartphone, or tablet device. In some embodiments, user-local computing device 1520 may be a portable computing device.
[0140] The servicing local computing device 1530 may be a computing device that serves the physical locality of the scientific instrument 1510 (e.g., any implementation of the computing device 1400 discussed herein). For example, the servicing local computing device 1530 may be a local device of the manufacturer of the scientific instrument 1510 or a third-party service company. In some implementations, the servicing local computing device 1530 may (e.g., via a direct communication path 1508 or via multiple “indirect” communication paths 1508, as discussed above) communicate with the scientific instrument 1510, the user local computing device 1520, and / or the remote computing device 1540 to receive data regarding the operation of the scientific instrument 1510, the user local computing device 1520, and / or the remote computing device 1540 (e.g., self-test results of the scientific instrument 1510, calibration coefficients used by the scientific instrument 1510, measurements of sensors associated with the scientific instrument 1510, etc.). In some implementations, the serving local computing device 1530 may (e.g., via a direct communication path 1508 or via multiple “indirect” communication paths 1508, as discussed above) communicate with the scientific instrument 1510, the user local computing device 1520, and / or the remote computing device 1540 to send data to the scientific instrument 1510, the user local computing device 1520, and / or the remote computing device 1540 (e.g., thereby updating programmed instructions (such as firmware) in the scientific instrument 1510 to initiate the execution of a test or calibration sequence in the scientific instrument 1510, updating programmed instructions (such as software) in the user local computing device 1520, or the remote computing device 1540, etc.). Users of scientific instrument 1510 may use scientific instrument 1510 or user local computing device 1520 to communicate with service local computing device 1530 to report problems with scientific instrument 1510 or user local computing device 1520, thereby requesting a technician to visit to improve the operation of scientific instrument 1510, order consumables or replacement parts associated with scientific instrument 1510, or for other purposes.
[0141] Remote computing device 1540 may be a computing device located remotely from scientific instrument 1510 and / or user local computing device 1520 (e.g., any embodiment of the embodiments of computing device 1400 discussed herein). In some embodiments, remote computing device 1540 may be included in a data center or other large-scale server environment. In some embodiments, remote computing device 1540 may include network-attached storage (e.g., as part of storage device 1504). Remote computing device 1540 may store data generated by scientific instrument 1510, perform analysis on data generated by scientific instrument 1510 (e.g., according to programmed instructions), facilitate communication between user local computing device 1520 and scientific instrument 1510, and / or facilitate communication between service local computing device 1530 and scientific instrument 1510.
[0142] In some implementation schemes, Figure 15 One or more components of the scientific instrument support system 1500 illustrated herein may be absent. Furthermore, in some embodiments, there may be... Figure 15 The scientific instrument support system 1500 comprises multiple elements among various components. For example, the scientific instrument support system 1500 may include multiple user local computing devices 1520 (e.g., different user local computing devices 1520 associated with different users or located in different locations). In another example, the scientific instrument support system 1500 may include multiple scientific instruments 1510, all of which communicate with a service local computing device 1530 and / or a remote computing device 1540; in such embodiments, the service local computing device 1530 may monitor these multiple scientific instruments 1510, and the service local computing device 1530 may cause updates, or other information may be simultaneously “broadcast” to multiple scientific instruments 1510. Different scientific instruments among the scientific instruments 1510 in the scientific instrument support system 1500 may be positioned close to each other (e.g., in the same room) or far from each other (e.g., on different floors of a building, in different buildings, in different cities, etc.). In some implementations, scientific instrument 1510 may be connected to an Internet of Things (IoT) stack that allows command and control of scientific instrument 1510 via web-based applications, virtual or augmented reality applications, mobile applications, and / or desktop applications. Any of these applications may be accessible to a user operating a user-local computing device 1520 that communicates with scientific instrument 1510 via an intermediate remote computing device 1540. In some implementations, scientific instrument 1510 may be sold by the manufacturer along with one or more associated user-local computing devices 1520 that are part of a local scientific instrument computing unit 1512.
[0143] In some such implementations, remote computing device 1540 and / or user local computing device 1520 may combine data from different types of scientific instruments 1510 included in scientific instrument support system 1500.
[0144] Furthermore, in at least one embodiment, a digital twin system may be implemented at one or more computing devices of the scientific instrument support system 1500. For example, the digital twin system may be deployed at a user-local computing device 1520, a remote computing device 1540, and / or a service-local computing device 1530. Regardless of location, the digital twin system may receive information and data from the scientific instrument 1510 and use this information and data to simulate one or more of the physical assets (e.g., scientific instrument 1510), the sample, and the interactions between the physical assets and the sample, using logic implemented at the scientific instrument support module, as referenced above. Figure 8 The scientific instrument support module is described.
[0145] The following paragraphs provide various embodiments of the implementation schemes disclosed herein:
[0146] In at least one embodiment, a scientific instrument support system includes: a sample; a physical asset configured to interact with the sample; and one or more computing devices storing executable instructions that, when executed, cause the one or more computing devices to generate one or more predictions based on simulations of one or more of the sample, the physical asset, or the interaction between the physical asset and the sample, and to update a physics-based digital model of the sample based on the accuracy of the one or more predictions, wherein the physics-based digital model is used to reconstruct the chemical and physical properties of the sample to generate simulations of the sample. In an embodiment including the foregoing embodiments, the executable instructions further cause the one or more computing devices to use the physics-based digital model to generate training data to train one or more neural networks to generate the one or more predictions, and wherein the one or more neural networks are implemented at the one or more computing devices. In an embodiment including any of the foregoing embodiments, the physics-based digital model of the sample is calibrated based on data obtained using one or more analytical techniques to set initial parameter values for the physics-based digital model, and based on a comparison of simulation results of the interaction between the physical asset and the sample with experimental results of the interaction between the physical asset and the sample. In any of the preceding embodiments, during operation of the physical asset, feedback from the controller of the physical asset is used to update the physics-based digital model during the operation of the physical asset. In any of the preceding embodiments, the accuracy of the one or more predictions is evaluated based on a comparison of data obtained from simulation results of operation of the physical asset with data obtained from experimental results of operation of the physical asset. In any of the preceding embodiments, the one or more predictions are generated based on a request received from an operator, and wherein the one or more predictions are further generated based on information about one or more properties and states of the physical asset, retrieved from the controller of the physical asset. In any of the preceding embodiments, the one or more predictions are output by one or more neural networks trained to infer optimal settings of the physical asset to be used during operation of the physical asset, and wherein the one or more neural networks are trained based on the simulation. In any of the preceding embodiments, the physical asset includes means configured to emit a beam of charged particles, and wherein the interaction between the physical asset and the sample includes the impact of the charged particle beam on the sample.
[0147] In at least one embodiment, a method includes: receiving, at one or more computing devices, a request to perform an operation on a sample using a physical asset; performing, at the one or more computing devices and in response to the request, one or more simulations of an interaction between the physical asset and the sample, at least in part based on a digital model of the sample; returning, at a user interface of the one or more computing devices, optimized settings for the physical asset determined based on the one or more simulations; and updating the digital model using the one or more computing devices based on a comparison of a result using the optimized settings with a predicted result. In an embodiment including the foregoing embodiments, receiving the request includes receiving sample information and / or a target result, and wherein each of the simulation of the operation and the execution of the operation using the physical asset is based on the target result. In an embodiment including any of the foregoing embodiments, updating the digital model includes: using the comparison to determine a difference between the result using the optimized settings and the predicted result, and adjusting the parameters of the digital model based on the difference. In an embodiment including any of the foregoing embodiments, the digital model is updated while the operation is performed using the physical asset, and wherein the digital model is updated based on a periodically obtained comparison of the experimental progress of the operation with the predicted progress of the operation. In an embodiment including any of the foregoing embodiments, the digital model includes one or more neural networks, and wherein the one or more neural networks are trained to perform the one or more simulations at least in part using simulations generated by a physics-based model configured to reconstruct the chemical and physical properties of the sample. In an embodiment including any of the foregoing embodiments, in response to receiving the request, the method further includes: acquiring information about the properties of the physical asset related to performing the operation, and predicting values to be used for the optimization settings, and wherein, while the physical asset performs the operation, the predicted values are adjusted in response to a detected deviation between the measured progress of the operation and the predicted progress of the operation. In an embodiment including any of the foregoing embodiments, the physical asset includes a focused ion beam microscope, and wherein the one or more simulations include removing material from the sample using a focused ion beam emitted by the focused ion beam microscope.
[0148] In at least one embodiment, a non-transitory computer-readable medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to: perform one or more simulations based on a requested result using a physical-based model of a sample and a physical-based model of a physical asset; predict settings to be used to operate the physical asset based on the one or more simulations; operate the physical asset using the predicted settings; compare the actual result of the operation of the physical asset with the requested result; and adjust the physical-based model of the sample using the comparison. In an embodiment including the preceding embodiment, the physical-based model of the physical asset is generated based on computer-aided design (CAD) drawings of at least a portion of the physical asset. In an embodiment including any of the foregoing embodiments, the physical-based model of the sample is generated based on sample information, the sample information including one or more of the sample's known material properties, the sample's chemical elemental profile, or the sample's compositional distribution, and wherein the sample information is obtained from one or more of an operator, a three-dimensional (3D) model of the sample, a database, a lookup table, or a text description. In an embodiment including any of the foregoing embodiments, the one or more simulations are performed based on the definition of a 3D space for performing the one or more simulations, the generation of objects interacting within the 3D space, and wherein the one or more simulations include modeling surface evolution during a milling process to be performed on the sample. In an embodiment including any of the foregoing embodiments, the physical-based model of the sample is implemented by one or more neural networks, which are trained offline to perform the one or more simulations, and wherein the comparison is used to update the weights of the one or more neural networks.
[0149] Although this disclosure has been described in detail with reference to specific embodiments, it will be apparent to those skilled in the art that various modifications and changes can be made without departing from the spirit and scope of this disclosure. Therefore, the specification and drawings are to be considered illustrative rather than restrictive.
Claims
1. A scientific instrument support system, comprising: sample; Physical assets configured to interact with the sample; and One or more computing devices, wherein executable instructions are stored on the one or more computing devices, and the executable instructions, when executed, cause the one or more computing devices to: One or more predictions are generated based on simulations of one or more of the sample, the physical asset, or the interaction between the physical asset and the sample; and The physics-based digital model of the sample is updated based on the accuracy of the one or more predictions, wherein the physics-based digital model is used to reconstruct the chemical and physical properties of the sample to generate the simulation of the sample.
2. The scientific instrument support system of claim 1, wherein the executable instructions further cause the one or more computing devices to: The physical-based digital model is used to generate training data to train one or more neural networks to generate the one or more predictions, and the one or more neural networks are implemented on the one or more computing devices.
3. The scientific instrument support system of claim 1, wherein the initial parameter values of the physics-based digital model are set based on data obtained using one or more analytical techniques, and the physics-based digital model of the sample is calibrated based on a comparison of simulation results of the interaction between the physical asset and the sample with experimental results of the interaction between the physical asset and the sample.
4. The scientific instrument support system of claim 1, wherein during operation of the physical asset, feedback from the controller of the physical asset is used to update the physics-based digital model during the operation of the physical asset.
5. The scientific instrument support system of claim 1, wherein the accuracy of the one or more predictions is evaluated based on a comparison of data obtained from simulation results of operations using the physical asset with data obtained from experimental results of operations using the physical asset.
6. The scientific instrument support system of claim 1, wherein the one or more predictions are generated based on a request received from an operator, and wherein the one or more predictions are further generated based on information about one or more properties and states of the physical asset, the information being retrieved from a controller of the physical asset.
7. The scientific instrument support system of claim 1, wherein the one or more predictions are output by one or more neural networks trained to infer optimal settings of the physical asset to be used during operation of the physical asset, and wherein the one or more neural networks are trained based on the simulation.
8. The scientific instrument support system of claim 1, wherein the physical asset includes means configured to emit a beam of charged particles, and wherein the interaction between the physical asset and the sample includes the impact of the charged particle beam on the sample.
9. A method comprising: At one or more computing devices, a request is received to perform operations on the sample using physical assets; At the one or more computing devices and in response to the request, one or more simulations of the interaction between the physical asset and the sample are performed, at least in part based on the digital model of the sample. At the user interface of the one or more computing devices, return the optimized settings for the physical asset determined based on the one or more simulations; as well as Using the one or more computing devices, the digital model is updated based on a comparison between the results using the optimized settings and the prediction results.
10. The method of claim 9, wherein receiving the request includes receiving sample information and / or a target result, and wherein each of the simulation of the operation and the execution of the operation using the physical asset is based on the target result.
11. The method of claim 9, wherein updating the digital model comprises: The comparison is used to determine the difference between the result obtained using the optimized settings and the prediction result, and the parameters of the digital model are adjusted based on the difference.
12. The method of claim 9, wherein the digital model is updated while the operation is performed using the physical asset, and wherein the digital model is updated based on a periodic comparison of the experimental progress of the operation with the predicted progress of the operation.
13. The method of claim 9, wherein the digital model comprises one or more neural networks, and wherein the one or more neural networks are trained to perform the one or more simulations at least in part using simulations generated by a physics-based model configured to reconstruct the chemical and physical properties of the sample.
14. The method of claim 9, wherein in response to receiving the request, the method further comprises: Information about the properties of the physical asset related to the execution of the operation is collected, as well as a predicted value to be used for the optimization settings, wherein, when the physical asset performs the operation, the predicted value is adjusted in response to a detected deviation between the measured progress of the operation and the predicted progress of the operation.
15. The method of claim 9, wherein the physical asset comprises a focused ion beam microscope, and wherein the one or more simulations comprise removing material from the sample using a focused ion beam emitted by the focused ion beam microscope.
16. A non-transitory computer-readable medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to: Using the physical-based model of the sample and the physical-based model of the physical assets, perform one or more simulations based on the requested results; Based on the one or more simulations, predict the settings to be used to operate the physical assets; Operate the physical assets using the predicted settings; The actual result of the operation on the physical asset is compared with the requested result; and The comparison is used to adjust the physical-based model of the sample.
17. The non-transitory computer-readable medium of claim 16, wherein the physical-based model of the physical asset is generated based on computer-aided design (CAD) drawings of at least a portion of the physical asset.
18. The non-transitory computer-readable medium of claim 16, wherein the physical-based model of the sample is generated based on sample information, the sample information including one or more of the known material properties of the sample, the chemical elemental profile of the sample, or the compositional distribution of the sample, and wherein the sample information is obtained from one or more of an operator, a three-dimensional (3D) model of the sample, a database, a lookup table, or a text description.
19. The non-transitory computer-readable medium of claim 16, wherein the one or more simulations are performed based on the definition of a 3D space for performing the one or more simulations, the generation of objects interacting within the 3D space, and wherein the one or more simulations include modeling surface evolution during a milling process to be performed on the sample.
20. The non-transitory computer-readable medium of claim 16, wherein the physics-based model of the sample is implemented by one or more neural networks, the one or more neural networks being trained offline to perform the one or more simulations, and wherein the comparison is used to update the weights of the one or more neural networks.
