On-site auxiliary image acquisition method and system using charged particle microscope
The problems of navigation and radiation dose are solved by using machine learning models in charged particle microscopy, and efficient atomic structure visualization and low-dose imaging are achieved.
Patent Information
- Application Number
- CN202380035040.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-02
- Filing Date
- 2023-06-02
- Publication Date
- 2024-11-29
AI Technical Summary
In charged particle microscopy, when high resolution atomic images are obtained, the navigation of the sample and high radiation dose navigation can lead to sample damage, and images with low signal-to-noise ratios are difficult to clearly display the atomic structure.
By using trained machine learning models, atomic structure probability images are generated based on initial images with low signal to noise ratio, atomic positions are located and enhanced images are generated, field-assisted navigation and low dose scanning are achieved, reducing radiation damage to the sample.
It realizes the generation of high signal-to-noise ratio images at low doses, reduces sample damage, and improves the visual clarity and navigation efficiency of atomic structures.
Smart Images

Figure CN119053987A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. patent application No. 17 / 831,147, filed on June 2, 2022, which is incorporated herein by reference in its entirety. Technical Field
[0003] The field relates to charged particle microscopy. Background Art
[0004] Charged particle microscopy involves the use of an accelerated beam of charged particles as a source of illumination. Example types of charged particle microscopy include transmission electron microscopy, scanning electron microscopy, scanning transmission electron microscopy, and ion beam microscopy.
[0005] Scanning transmission electron microscopy (also known as STEM) can be used to collect high-resolution images of a sample at the atomic scale. In STEM imaging, an electron beam is scanned over a sample or a region of a sample. The electrons interact with the sample, causing elastically scattered electrons to leave the sample. In transmission imaging mode, the electrons transmitted through the sample are detected and used to form a microscopic image of the sample.
[0006] STEM atomic spectroscopy can provide deep insight into the structure of a material. However, obtaining atomic images with sufficient resolution for structural analysis of a material can be challenging. In a typical atomic spectroscopy workflow, the operator navigates around the sample to find a region of interest on the sample. When the region of interest is found, a final image of the region of interest is taken. Navigation involves scanning the electron beam across the sample or a region of the sample. In order to obtain an atomic resolution image, the electron beam should be aligned with the atomic columns in the irradiated area of the sample, which may require tilting the sample relative to the electron beam path to find the desired alignment. Many materials of interest have a limited electron dose budget. For these materials, lengthy navigation to find the region of interest and / or the use of high radiation doses (e.g., ) may result in damage to the sample before the final image is acquired. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 is a schematic diagram of a system for imaging a sample using a charged particle microscope according to an example.
[0008] Figure 2 is a block diagram of an example sample imaging application according to an example.
[0009] Figure 3A A low signal-to-noise ratio (SNR) image acquired using a charged particle microscope system according to an example is illustrated.
[0010] Figure 3BThe example is illustrated by a trained machine learning network based on Figure 3A The predicted image is generated from the low SNR image.
[0011] Figure 3C The example based on Figure 3B The enhanced image generated by the predicted image.
[0012] Figure 4A is a flow chart illustrating a method of navigating around a sample using a charged particle microscope, according to an example.
[0013] Figure 4B is an example Figure 4A Flowchart of a portion of the method depicted.
[0014] Figure 5 According to the example, Figure 2 An example graphical user interface presented by a sample imaging application depicted in FIG.
[0015] Figure 6 is a diagram of a specific implementation of a Cycle Generative Adversarial Network (CycleGAN) according to an example. DETAILED DESCRIPTION
[0016] General considerations
[0017] The present subject matter is described using specific implementations and examples. In some cases, as will be appreciated by those skilled in the art, the disclosed specific implementations and examples may be practiced without one or more of the specific details disclosed, or may be practiced with other methods, structures, and materials not specifically described herein. All specific implementations and examples described herein and shown in the drawings may be combined without any restriction to form any number of combinations, unless the context clearly dictates otherwise, such as if the proposed combination involves incompatible or mutually exclusive elements. The order of the actions in any process described herein may be rearranged, unless the context clearly dictates otherwise, such as if one action requires the result of another action as input.
[0018] For simplicity, and for continuity of description, the same or similar reference numerals may be used for the same or similar elements in different drawings, and when an element in one drawing appears with the same or similar reference numerals in other drawings, the description of the element will be deemed to be continued. In some cases, the term "corresponding to" may be used to describe the correspondence between elements of different drawings. In example usage, when an element in a first drawing is described as corresponding to another element in a second drawing, the element in the first drawing is deemed to have the characteristics of the other element in the second drawing, and vice versa, unless otherwise specified.
[0019] The word "include" and its derivatives, such as "comprises" and "containing", are to be interpreted in an open, inclusive sense, i.e., as "including but not limited to". Unless the context indicates otherwise, the singular forms "a", "an", "at least one", and "the" include plural referents. When used between the last two elements of a list of elements, the term "and / or" refers to any one or more of the listed elements. The term "or" is generally used in its broadest sense, i.e., to mean "and / or", unless the context clearly indicates otherwise. When used to describe a range of sizes, the phrase "between X and Y" means a range that includes X and Y. As used herein, "device" may refer to any individual device, a collection of devices, a part of a device, or a collection of parts of a device.
[0020] Overview
[0021] The subject matter disclosed herein relates to imaging a sample with a charged particle microscope under imaging conditions that can produce acquired images with a low signal-to-noise ratio (SNR). In these low SNR images, the atomic structure in the imaged region of the sample is not clearly visible to the naked eye due to the low resolution of the image. The methods and systems disclosed herein can use low SNR images acquired using a charged particle microscope to generate enhanced images that indicate information about the atomic structure in the imaged region of the sample. The enhanced image can be displayed in a user interface to provide an operator with on-site assistance while navigating on the sample to find critical areas of the sample (e.g., inhomogeneous structures), or during in situ experiments to capture dynamic effects in the sample composition.
[0022] In some examples, the methods and systems disclosed herein may be configured to obtain image data of a region of a sample, generate one or more initial images based on the image data, and generate one or more enhanced images indicating information about atomic structures in the region of the sample. In various examples, the generation of the one or more enhanced images includes using a trained machine learning model that accepts the initial image as input to predict atomic structure probabilities in the sample. The predicted atomic structure probabilities can be used to detect atomic positions in the region of the sample.
[0023] In various examples, imaging conditions that produce an acquired image with a low SNR may include a low dose of a charged particle beam, a short dwell time (or a fast scan speed) of a charged particle beam, or a sparse scan. Thus, the methods and systems disclosed herein enable field-assisted low charged particle beam doses and / or fast scans and / or sparse scans of a sample. In some examples, the methods and systems may enable the sample to be imaged using a charged particle beam dose that is significantly lower than the dose required to acquire a high-resolution image of the sample. Advantageously, an area of the sample may be repeatedly scanned using a low dose during field assistance before cumulative irradiation using a low dose would be equivalent to irradiation using a single high-dose scan. The low dose may also help avoid scenarios where the charged particle beam changes the dynamics of structures in the sample while capturing an image of the sample using field assistance.
[0024] Example - Sample Imaging System with Field Assist
[0025] Figure 1 An example of a system 100 for imaging a sample with field assistance is illustrated. The system 100 includes a charged particle microscope system 104 for studying and analyzing a sample. The system 100 may also include a computing environment 116 having a sample imaging application 200. In the illustrated example, the charged particle microscope system 104 includes a charged particle microscope 108 enclosed in a vacuum chamber 120 and a controller 112 that may be inside or outside the vacuum chamber 120. The controller 112 is communicatively coupled to the computing environment 116.
[0026] The controller 112 is connected to the various components of the charged particle microscope 108 and can transmit control / power signals to the components and receive data from the components. The controller 112 can allow the operation of the charged particle microscope system 104 to be controlled from the sample imaging application 200, as well as allow image data collected by the charged particle microscope 108 to be transmitted to the computing environment 116 and used by the sample imaging application 200. The controller 112 can be implemented in any suitable combination of hardware and software.
[0027] Any type of charged particle microscope 108 suitable for acquiring images of a sample may be used in the system 100. For purposes of illustration, the charged particle microscope 108 is depicted as a scanning transmission electron microscope (STEM), but the charged particle microscope 108 is not limited to the particular STEM configuration depicted. The STEM may be operated in a STEM mode (i.e., where a charged particle beam is scanned over an area of the sample) or in a transmission electron microscope (TEM) mode (i.e., without a scanning beam). Other examples of microscopes may include, but are not limited to, cryogenic electron microscopes (cryo-EM), ion-based microscopes, and proton microscopes.
[0028] The charged particle microscope 108 includes a first electro-optical system 144 defining an optical axis 124. A sample 128 to be studied and / or analyzed may be positioned below the first electro-optical system 144 and along the optical axis 124. The sample 128 is supported by a sample holder 132 (or stage), which in some examples may have the ability to translate, rotate, and / or tilt the sample. The controller 112 may be connected to the sample holder 132 to provide sample position control signals to the sample holder 132. The sample holder 132 may allow different areas of the sample to be positioned and / or tilted relative to the optical axis 124 (e.g., during navigation on the sample to find an area of interest on the sample).
[0029] The charged particle microscope 108 includes a charged particle source 136 positioned above a first electro-optical system 144. The charged particle source 136 can be, for example, an electron source (e.g., a Schottky gun), a positive ion source (e.g., a gallium ion source or a helium ion source), a negative ion source, a proton source, or a positron source. The charged particle source 136 generates a charged particle beam 140. The first electro-optical system 144 receives the charged particle beam 140 and arranges the charged particle beam 140 into a field of view on the sample 128.
[0030] The first electro-optical system 144 may include one or more electro-optical components. In some examples, the electro-optical components may be connected to the controller 112 to allow the electro-optical power of the components to be set and adjusted by the controller 112. For illustrative purposes, the first electro-optical system 144 may include focusing lenses 148a, 148b, a focusing stigmator 148c, and a focusing aperture 148d. The first electro-optical system 144 may include a scanning coil 150, which is operable to scan the charged particle beam 140 over an area of the sample 128. In other examples, the first electro-optical system 144 may have other components or fewer components than those illustrated.
[0031] When the charged particle beam 140 is incident on the sample 128, the charged particles in the beam interact with structures (e.g., atoms) in the sample 128, causing various types of radiation to be emitted from the sample 128. For example, when the charged particle beam 140 is an electron beam, the radiation emitted from the sample can include any combination of Auger electrons, secondary electrons, X-rays, backscattered electrons, cathodoluminescence, loss electrons, transmitted electrons, and diffracted electrons. The various types of radiation can be detected and used to form one or more images of the sample.
[0032] In some examples, the charged particle microscope 108 may include one or more detector systems to capture images of the sample 128 using one or more detector modalities. In one example, the charged particle microscope 108 may include a first detector system 152 to collect image data from the sample 128. The first detector system 152 may include a STEM detector, such as a bright field detector, annular bright field detector, dark field detector, annular dark field detector, high angle annular dark field (HAADF) detector, segmented STEM detector, or integrated differential phase contrast (iDPC) detector. In a specific example, the first detector system 152 may include a HAADF detector that detects charged particles (e.g., electrons) scattered to higher angles. In another specific example, the first detector system 152 may include an annular bright field detector and annular dark field detector to simultaneously capture bright field images and dark field images of the sample.
[0033] In one example, the charged particle microscope 108 may include one or more additional detector systems, such as a second detector system 156 and a third detector system 160, to obtain additional image data from the sample. For example, the detector systems 156, 160 may be spectroscopic systems. For illustrative purposes, the second detector system 156 is shown as being positioned below the sample 128, while the third detector system 160 is shown as being positioned above the sample 128. In a specific example, the second detector system 156 may include electron energy loss spectroscopy, and the third detector system 160 may include energy dispersive X-ray spectroscopy.
[0034] The charged particle microscope 108 may include a second optical system 164 to direct the charged particles transmitted through the sample 128 into the field of view of the detector systems 152, 156. For purposes of illustration, the second optical system 164 may include an objective lens 168a, an objective lens anastigmator 168b, and an objective lens aperture 168c. In other examples, the second optical system 164 may have other components or fewer components than those depicted.
[0035] The computing environment 116 may have any suitable configuration for running the sample imaging application 200. For example, the computing environment 116 may include a processor 180, a memory 184, a display device 188, and a data storage device 192. The sample imaging application 200 may be loaded into the memory 184 and executed by the processor 180. The sample imaging application 200 may present a user interface on the display device 188, and may present a sample image in the user interface and collect microscope control settings from the user interface. The sample imaging application 200 may provide the microscope control settings to the controller 112 and receive detector data from the controller 112. The computing environment 116 may include other components not specifically shown, such as input devices, output devices other than the display device 188, communication connections, other memories other than the memory 184, and other processors other than the processor 180.
[0036] Example - Sample Imaging Application
[0037] Figure 2 An example of a sample imaging application 200 is illustrated. The sample imaging application 200 may include imaging logic 204, atomic structure prediction logic 208, atomic position detection logic 210, and user interface logic 216. The imaging logic 204 may perform various functions related to acquiring an image of a sample. The atomic structure prediction logic 208 may perform various functions related to predicting atomic structure probabilities in the acquired sample image. The atomic position detection logic 210 may perform various functions related to finding atomic positions in the predicted image including atomic structure probabilities. The user interface logic 216 may perform various functions related to presenting the acquired and enhanced image of the sample to a user and receiving input from the user. The sample imaging application 200 may have other logical components and data structures that are not specifically illustrated.
[0038] The imaging logic 204 may perform various functions related to acquiring images of a sample using a charged particle microscope system. During navigation over the sample, the imaging logic 204 may cause the sample to be placed at different locations relative to the field of view of the charged particle beam. For example, the imaging logic 204 may provide a controller (e.g., Figure 1 In some examples, imaging logic 204 may derive the position information of the sample based on user-defined settings received from user interface logic 216. The controller may then provide position information to a sample holder (e.g., Figure 1 The sample holder 132 in the embodiment of the present invention provides a control set point to translate and / or tilt the sample. In some cases, if a charged particle microscope (e.g., Figure 1 If the charged particle microscope 108 in the embodiment of the present invention is mounted on a stage, the charged particle microscope can be translated and / or tilted relative to the sample to achieve a desired relative positioning between the sample and the scanning field of view of the charged particle beam.
[0039] The imaging logic 204 may enable different types of images of the sample to be captured by the charged particle microscope system. For example, the imaging logic 204 may provide a controller of the charged particle microscope system with detector systems that should be activated during imaging of a region of the sample. In one example, the imaging logic 204 may enable images of the sample to be acquired using one detector modality (e.g., a HAADF detector modality) or at least two detector modalities (e.g., a brightfield and darkfield detector modality or a HAADF detector modality and a spectral detector modality).
[0040] The imaging logic 204 may receive detector data from the charged particle microscope system. In some examples, the imaging logic 204 may construct an initial image of the sample based on the detector data. For example, as the charged particle beam scans across an area of the sample (e.g., in a raster pattern), the charged particle microscope detector system generates an output for each (x, y) scan beam position and tilt angle of the sample. The detector output for each scan beam position may provide information for a pixel of the sample image. The imaging logic 204 may use the detector output to construct one or more initial images of the sample. In some examples, the imaging logic 204 may apply a timestamp to the constructed initial image and store the initial image in a data storage device (e.g., Figure 1 The data storage device 192 is stored in the sample imaging application 200 for use by other components of the sample imaging application 200.
[0041] Atomic structure prediction logic 208 may perform various functions related to predicting atomic structure probabilities in an initial image generated by imaging logic 204. In some examples, the initial image may have a low SNR. Figure 3A An example of a low SNR image 300 acquired by a HAADF detector from a region of a sample is shown. In some examples, the atomic structure prediction logic 208 may acquire one or more low SNR images, such as Figure 3A 300 shown in FIG. 1 and predicting atomic structure probabilities in regions of the sample corresponding to the one or more low SNR images. The output of the atomic structure prediction logic 208 may be an atomic structure probability image, which is an image in which there is detectable contrast between regions of the image that may include atomic structure and regions of the image that may not include atomic structure.
[0042] In some examples, the atomic structure prediction logic 208 uses a trained machine learning model to predict atomic structure probabilities. For example, the trained machine learning model can be a neural network (e.g., a convolutional neural network) that has been trained to predict atomic structure probabilities. In some examples, the atomic structure prediction logic 208 may generate an inference request including a set of one or more initial images (or acquired images). The set of one or more initial images may form an input vector for the trained neural network. In some examples, the set of one or more initial images may be images acquired using the same detector modality. In other examples, the set of one or more initial images may be images acquired using different detector modalities. The atomic structure prediction logic 208 may transmit the inference request to the inference engine 220 including the trained machine learning model 224. The inference engine 220 may be in the same computing environment as the sample imaging application 200, or may be in a different computing environment (e.g., on an AI server in the cloud).
[0043] Upon receiving an inference request, the inference engine 220 applies the set of one or more initial images in the inference request to the input of the trained machine learning model 224 to obtain a prediction. The prediction may be an image including atomic structure probabilities. Figure 3B Shown based on Figure 3A An example of a predicted image 310 output by a trained neural network for a low SNR image 300 is shown (pixel intensities in areas of the image 310 that are likely to include atomic structures are higher than pixel intensities in areas of the image that are unlikely to include atomic structures). Inference engine 220 generates an inference that includes the predicted image. Inference engine 220 may add other information to the inference, such as a confidence in the neural network's prediction and / or an explanation of the neural network's prediction. Inference engine 220 returns the inference to atomic structure prediction logic 208.
[0044] The confidence in the predictions made by the trained machine learning model 224 can be determined using the same techniques that exist for validation of artificial neural networks. For example, the trained machine learning model 224 can be applied to a low quality image (or low SNR image), and the predictions of the trained machine learning model 224 can be compared to a high quality image (or high SNR image) to determine the confidence in the prediction. The low quality images and high quality images can be generated using different imaging conditions, or simulated by artificially degrading a high quality image from measurement or from simulation.
[0045] The atom position detection logic 210 may extract the atom positions from the predicted image including the atomic structure probabilities. In some examples, the atom position detection logic 210 uses image segmentation to extract the atom positions from the atomic structure probability image. Various types of image segmentation techniques may be used. One technique may be based on threshold processing. In threshold processing, a pixel intensity threshold is set to classify pixels in the atomic structure probability image into atomic structure pixels and background pixels. The atom position detection logic 210 may output the atom positions or may output an enhanced image indicating the atom positions. In some examples, an enhanced image may be generated by superimposing the atom positions on the predicted image. Figure 3C Shown includes superimposed Figure 3B 2. An example of an enhanced image 320 of atomic positions determined by the atom position detection logic 210 on the predicted image 310 in FIG. 4 (large dots indicate atomic positions).
[0046] User interface logic 216 may execute and display the display device (e.g., Figure 1 The user interface logic 216 may be configured to display a user interface on a display device 188 in the display device 188, respond to events triggered from the user interface, and collect various functions related to user input from the user interface. For example, the user interface logic 216 may receive an image from the atomic structure prediction logic 208 and / or the atomic position detection logic 210. The user interface logic 216 may cause a graphical representation of at least a portion of the image to be presented in a specified area of the user interface. In one example, when navigating on a sample or scanning an area of a sample during an in-situ experiment, the user interface logic 216 may present an image sequence in the user interface. The image sequence may include one or more acquired images of the sample area and one or more enhanced images generated based on the acquired images.
[0047] User interface logic 216 may receive input entered by a user at the user interface. In one example, the input may be settings for a charged particle microscope system or annotations to an image presented in the user interface. In some cases, the user may select a control on the user interface to begin imaging an area of the sample. In this case, user interface logic 216 may receive a request to begin imaging and transmit the request to imaging logic 204. User interface logic 216 may cause other information to be displayed in the user interface, such as one or more metrics related to the operation of the trained machine learning model in inference engine 220.
[0048] In some cases, the sample imaging application 200 may include training logic 214, which may generate a request to train or retrain a machine learning model to predict atomic structure probabilities from low SNR images. In one example, the training logic 214 may generate a training request and transmit the training request to the training engine 230 to train the machine learning model 228.
[0049] The training engine 230 may receive a training request from the training logic 214 and train the machine learning model 228 to perform the task of predicting atomic structure probabilities in low SNR images. In some examples, the training engine 230 may be in the same computing environment as the sample imaging application 200. In other examples, the training engine 230 may be in a remote machine learning environment in the cloud. After training the machine learning model 228, the training engine 230 may notify the training logic 214. In response to the notification, the training logic 214 may deploy the trained machine learning model 224 (resulting from the training of the machine learning model 228) to the inference engine 220 for making predictions.
[0050] The sample imaging application 200 may be stored in one or more computer-readable storage media or computer-readable storage devices and executed by one or more processors (e.g., Figure 1 The computer readable medium may be executed by a processor 180 in the computing environment 116 shown. Any of the computer readable media herein may be non-transitory (e.g., volatile memory such as DRAM or SRAM, non-volatile memory such as magnetic storage devices, optical storage devices, etc.) and / or tangible.
[0051] Example - Training Dataset for Machine Learning
[0052] The training engine 230 may receive a training request from the sample imaging application 200. The training request may include a training data set 232, or the training engine 230 may retrieve the training data from a data storage device (e.g., Figure 1 The training data set 232 may be retrieved from the data storage device 192 in the training engine 230, or the training data set 232 may be generated by the training engine 230. The training data set 232 may include any combination of simulated images, real images, and acquired images of atomic structures. Preferably, the training data set 232 includes images from conditions similar to the target usage and materials (e.g., noise statistics and horizontal and atomic grid structures).
[0053] In one example, the training data set 232 may include low SNR images of the sample collected using a charged particle microscope system. Each low SNR image may be labeled with the name of the sample and the area on the sample where the low SNR image was obtained. The training data set may also include high SNR images of the sample collected using a charged particle microscope system. Each high SNR image may be labeled with the name of the sample and the area on the sample where the high SNR image was obtained. Using matching sample names and area labels as pairing criteria, input-output pairs of the training data set 232 may be generated by pairing each low SNR image (as input) with one of the high SNR images (as output). In some cases, two or more low SNR images (as input) collected using different detector modalities may be paired with one of the high SNR images (as output). In some cases, instead of collecting low SNR images directly from the charged particle microscope system, the collected high SNR images may be artificially degraded to form low SNR images used in the training data set.
[0054] In another example, the training data set 232 may include simulated images of atomic structures generated based on an atomic structure model and a noise model. The machine learning model may be trained and executed to generate simulated images. An example of a technique for generating simulated images is described in Lin, R., Zhang, R., Wang, C. et al., TEMImageNet training library and AtomSegNet deep learning model for high-precision atomic segmentation, localization, denoising and deblurring of atomic resolution images, Scientific Reports (SciRep) 11, 5386 (2021).
[0055] In another example, the training dataset may include real images obtained from an open access electron microscopy dataset, such as the Warwick electron microscopy dataset, available at github.com / Jeffrey-Ede / datasets / wiki and described in Jeffrey M Ede, “Warwick Electron Microscopy Datasets,” 2020 Mach. Learn. Sci. Technol. 1045003.
[0056] Example - Training a Machine Learning Model
[0057] In one example, the training engine 230 uses supervised training cycles interleaved with unsupervised training cycles to train the machine learning model 228. In one example, several supervised cycles (e.g., two or more supervised cycles) may be followed by several unsupervised cycles (e.g., two or more unsupervised cycles).
[0058] In one example, supervised and unsupervised training can use the Cycle Generative Adversarial Network (CycleGAN) model architecture (see Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros, “Unpaired Image-to-Image Translationusing Cycle-Consistent Adversarial Networks,” arXiv:1703.10593v7, August 24, 2020).
[0059] Figure 6 The CycleGAN architecture is illustrated, which includes two generators G and F. Generator G maps data from domain X to Y. Generator F maps data from domain Y to X. The CycleGan architecture includes two discriminators Dx and Dy. Discriminator Dy forces generator G to transform X into an output that is indistinguishable from domain Y. Discriminator Dx forces generator F to transform Y into an output that is indistinguishable from domain X. Discriminators Dx and Dy are only used during unsupervised training.
[0060] Training includes a forward loop and a reverse loop. In the forward loop, generator G receives input x and produces output Y°, and generator F receives Y° as input and produces output x°. The input x and output x° should be similar. The difference between x and x° is the cycle consistency loss in the forward direction and can be included in the optimization of the network. In the reverse loop, generator G receives input y and produces output X°, and generator G receives X° as input and produces output y°. The input y and output y° should be similar. The difference between y and y° is the cycle consistency loss in the reverse direction and can be included in the optimization of the network.
[0061] Discriminators Dx and Dy are used for unsupervised training. In the forward loop, discriminator Dy takes Y° and Y as input and produces an output representing real or fake. In the reverse loop, discriminator Dx takes X° and X as input and produces an output representing real or fake. In unsupervised training, input images x and y are unpaired and noisy images (e.g., low SNR images). Images X and Y are real images (e.g., high SNR images).
[0062] During supervised training, the discriminators Dx and Dy are not used. The G network is run independently of the larger structure using simulated images as input x and paired real images as labels y. The F network is also run independently of the larger structure using real images as input y and paired simulated images as labels x.
[0063] CycleGAN alternates between supervised and unsupervised training modes after a few cycles in one of the modes.
[0064] The generator and discriminator can be convolutional neural networks. In one example, a U-Net architecture can be used for generators G and F. U-Net is a convolutional neural network developed for biomedical image segmentation (see Olaf Ronneberger et al., "U-Net Convolutional Networks for Biomedical Image Segmentation", arXiv:1505.04597, May 18, 2015). In a specific example, a parameterized U-Net structure with a 256×256 input with 8 multi-resolution layers can be used for the generator. Other types of convolutional neural networks (such as the VGG architecture or SUNet: Swin Transformer UNet for Image Denoising) can be used for the generator. In one example, the discriminator can be a PatchGAN discriminator, which is a convolutional network in which the input image is mapped to an N×N array instead of a single scalar vector.
[0065] Example - Sample imaging using field-assisted methods
[0066] Figure 4A is a flow chart illustrating an exemplary method 400 of imaging a sample using a charged particle microscope system according to an example. Figure 4B is a block diagram illustrating a portion of method 400. Method 400 may be used Figure 1 The system depicted and / or Figure 2 The sample imaging described in the example should be performed. Figure 4A and Figure 4B Although illustrated once in a particular order, the operations may be reordered and / or repeated as needed and appropriate (eg, different operations illustrated as being performed sequentially may be suitably performed in parallel).
[0067] At 410, the method includes navigating to a region of interest (ROI) on the sample. For example, the sample can be positioned so that the ROI is within a field of view of a charged particle beam output by a charged particle microscope system. Positioning the sample can include (e.g., by a controller of the charged particle microscope system) transmitting appropriate controls to a sample holder to adjust the position of the sample relative to the charged particle beam.
[0068] At 420, the method includes setting imaging conditions for the sample. The imaging conditions may be set by adjusting one or more imaging parameters of the charged particle microscope system. For example, the method may include setting a dose of a charged particle beam applied to the sample. If the image is to be generated by scanning the ROI, the method may include setting a dwell time or a scan speed or a scan pattern of the charged particle beam. The method may also include activating a detector system for collecting image data. The method may also include adjusting a tilt angle of the sample. The parameter settings may come from a sample imaging application and may be applied by a controller of the charged particle microscope system.
[0069] At 430, the method includes acquiring image data from the ROI under the imaging conditions set in operation 420. For example, when the charged particle beam is incident on the ROI, the activated detector system may collect image data generated by the interaction between the charged particle beam and the ROI. In some examples, the ROI may be scanned by sweeping the charged particle beam across the ROI in a raster pattern (or another scanning pattern). The activated detector system may capture image data while scanning the ROI. In some examples, a single detector mode (e.g., a HAADF detector mode) may be used to capture image data. In other examples, at least two different detector modes (e.g., dark field and bright field detector modes or diffraction detector modes (e.g., HAADF detector mode) and spectral detector modes (e.g., electron energy loss spectroscopy or energy dispersion X-ray spectroscopy)) may be used to capture image data. In other examples, a camera may capture an image of the sample. The camera may include, for example, a CCD (charge coupled device) imaging sensor, a CMOS (complementary metal oxide semiconductor) imaging sensor, or more generally, a photodetector array. The camera can be operated in "movie" mode to capture a sequence of images of the sample.
[0070] At 440, the method includes generating a set of one or more initial images (or acquired images) based on the image data acquired in operation 430. For example, the image data may include the charged particle intensity measured at each scanning position of the charged particle beam. Each intensity may correspond to a pixel of the initial image. The initial image may be constructed as a set of pixels, where the pixel coordinates are related to the scanning position of the beam position, and the pixel value is determined by the measured intensity corresponding to the scanning position of the beam. In some examples, the imaging conditions set in operation 420 produce an initial image with a low SNR (i.e., a low SNR image).
[0071] At 450, the method includes using the trained machine learning model to generate a predicted image indicating atomic structure probabilities in the ROI. In some examples, the trained machine learning model is a neural network (e.g., a convolutional neural network) that has been trained to predict atomic structure probabilities based on one or more low SNR images. The output of the neural network can be a predicted image indicating atomic structure probabilities. In some examples, one or more of the initial images generated in operation 440 can be applied to the input of the trained machine learning model to obtain a predicted image. For illustration purposes, Figure 4B A low SNR image 460 is depicted applied to the input of the trained machine learning model 224, which then generates a predicted image 464 indicating atomic structure probabilities. In some cases, two or more low SNR images may be applied to the input of the trained machine learning model 224 (e.g., two or more low SNR images captured with different detector modalities), and the trained machine learning model 224 may output a predicted image based on these two or more low SNR images.
[0072] At 470, the method includes locating atomic positions in the predicted image generated in operation 450. In one example, image segmentation can be used to locate atomic positions. Various types of image segmentation can be used. One example of an image segmentation technique is thresholding. In thresholding, pixel intensity thresholds are set to classify pixels in the predicted image as atomic pixels and background pixels. The method may include generating an enhanced image by superimposing atomic objects (e.g., geometric shapes representing atoms) on the predicted image at the atomic positions found by image segmentation. For illustration purposes, Figure 4B A predicted image 464 is shown as input to the atom position detection logic 210 . Figure 4B An enhanced image 468 is shown that reveals the atomic positions (large dots 469) in the predicted image 464.
[0073] At 480, the method may include presenting an image of the sample on a display in real time. In some examples, the method may include recording the image for playback after image acquisition. Any combination of the images generated in operations 440, 450, and 470 may be presented and / or recorded. Figure 4BIt is shown that any one of images 460, 464, and 468 can be provided to the user interface logic 216, which can update the user interface with the image in real time. In one example, an initial image (e.g., a low SNR image) can be presented, followed by a predicted image generated based on the initial image, followed by an enhanced image generated based on the predicted image. In some examples, the predicted image can be displayed together with the confidence of the prediction. The image can be presented in the user interface to allow the user to interact with the image. For example, the user can zoom in on a portion of the enhanced image and select or mark the portion of the image for further investigation. In another example, the user can detect that the charged particle beam is not properly aligned with the atomic column in the ROI, and can determine the adjustment to the tilt of the sample based on the image.
[0074] At 485, the method may include determining whether to reacquire image data from the ROI. For example, the method may determine whether to reacquire image data from the ROI based on user input. For example, if the user detects that the sample is not properly aligned with the charged particle beam, the user may adjust the setting of the sample tilt angle through the user interface and trigger the reacquisition of image data of the ROI. In another example, if the user detects that the charged particle beam dose is too small or too large, the user may adjust the imaging conditions and trigger the reacquisition of image data of the ROI. If the method determines that the image data is to be reacquired, the method may reacquire the image data by returning to operation 420 and adjusting the imaging conditions (e.g., based on user input). If the method determines that the image data of the ROI does not need to be reacquired, the method may continue to operation 490.
[0075] At 490, the method includes acquiring a final image of the ROI. The final image may be acquired under imaging conditions that are different from the imaging conditions used to acquire the initial image of the ROI in operation 420. Specifically, the imaging conditions used when acquiring the final image may produce a final image with a higher SNR than the initial image. In some examples, a charged particle beam dose used when acquiring the final image may be higher than a charged particle beam dose used when acquiring the initial image, and / or a scanning speed used when acquiring the final image may be lower than a scanning speed used when acquiring the initial image, and / or a scanning pattern used when acquiring the low SNR image may be sparser than a scanning pattern used when acquiring the final image.
[0076] At 495, the method may include determining whether to navigate to another ROI on the sample. If the method determines that another ROI on the sample should be processed, the method may return to operation 410 to navigate to the new ROI. In some cases, the user may indicate through the user interface that another ROI should be processed. As part of indicating that another ROI should be processed, the user may specify a new ROI. Alternatively, the method may automatically determine a new ROI to be processed based on a predetermined experiment or navigation plan.
[0077] At 499, if the method determines that processing of another ROI is not required, image acquisition may be terminated.
[0078] Example - Graphical User Interface
[0079] Figure 5 An example graphical user interface (GUI) 500 that may be presented by a sample imaging application on a display device is depicted. A user may interact with GUI 500 using any suitable input device and input technology (e.g., movement of a cursor, motion capture, facial recognition, gesture detection, voice recognition, actuation of a button, etc.).
[0080] GUI 500 may include a data display area 502 , a data analysis area 504 , a charged particle microscope control area 506 , and a settings area 508 . Figure 5 The specific number and arrangement of regions depicted in GUI 500 are illustrative only, and any number and arrangement of regions may be included in GUI 500, including any desired features.
[0081] The data display area 502 may display an image generated by a sample imaging application. For example, the data display area 502 may display any one of a low-resolution structural image, a structural position image, a structural object image, and a structural type image.
[0082] Data analysis area 504 may display the results of the data analysis. For example, data analysis area 504 may display a region of interest indicated by a user in an image displayed in the data display area. In some examples, data display area 502 and data analysis area 504 may be combined in GUI 500.
[0083] The charged particle microscope control area 506 may include options that allow a user to control the charged particle microscope system. For example, the charged particle microscope control area 506 may include user-selectable options that allow adjustment of parameters of the charged particle microscope system.
[0084] Settings area 508 may include options that allow a user to control features and functions of GUI 500 and / or perform other computing operations with respect to data display area 502 and data analysis area 504 (eg, saving data on a storage device).
[0085] In view of the above specific implementation and examples of the disclosed subject matter, the present application discloses the additional embodiments listed below. It should be noted that one feature of a separate embodiment or one or more features of an embodiment adopted in combination and optionally in combination with one or more features of one or more other embodiments are also additional embodiments of the disclosure of the present application.
[0086] Embodiment 1 is a method for imaging a sample using a charged particle microscope system, the method comprising: acquiring one or more first images of a region of the sample under first imaging conditions using the charged particle microscope system; applying the one or more first images to the input of a trained machine learning model to obtain a predicted image indicating the probability of an atomic structure in the region of the sample; and in response to obtaining the predicted image, displaying an enhanced image indicating the positions of atoms in the region of the sample based on the atomic structure probability in the predicted image.
[0087] Embodiment 2 includes the subject matter described in Embodiment 1, and also includes using the charged particle microscope system to acquire a second image of the area of the sample under second imaging conditions; wherein the second imaging condition is selected based on the predicted image or the enhanced image so that the signal-to-noise ratio of the second image is higher than the signal-to-noise ratio of the one or more first images.
[0088] Embodiment 3 includes the subject matter of Embodiment 2, and further provides for triggering acquisition of the second image of the region of the sample from a user interface after displaying the enhanced image.
[0089] Embodiment 4 includes the subject matter of any one of Embodiments 2 to 3, and further provides that the charged particle beam dose under the first imaging condition is lower than the charged particle beam dose under the second imaging condition.
[0090] Embodiment 5 includes the subject matter of any one of Embodiments 2 to 4, and further provides that a first scanning pattern used in acquiring the one or more first images is sparse compared to a second scanning pattern used in acquiring the second image.
[0091] Embodiment 6 includes the subject matter of any one of Embodiments 1 to 5, and further includes sequentially displaying the one or more first images and at least one of the predicted image and the enhanced image on a user interface.
[0092] Embodiment 7 includes the subject matter of any one of Embodiments 1 to 6, and further provides that acquiring the one or more first images of the region of the sample includes scanning a charged particle beam over the region of the sample.
[0093] Embodiment 8 includes the subject matter described in Embodiment 7, and further provides that acquiring the one or more first images of the area of the sample also includes collecting image data from the area of the sample using a single detector modality and constructing the one or more first images based on the collected image data.
[0094] Embodiment 9 includes the subject matter according to embodiment 8, and further provides that the single detector modality is a high-angle annular dark field detector modality.
[0095] Embodiment 10 includes the subject matter described in Embodiment 7, and further provides that acquiring the one or more first images of the region of the sample includes collecting image data from the region of the sample using at least two different detector modalities and constructing the one or more first images based on the collected image data.
[0096] Embodiment 11 includes the subject matter described in embodiment 10, and further provides that the at least two different detector modalities include one or more of the following: a dark field detector modality, an annular dark field detector modality, a bright field detector modality, a high-angle annular dark field detector, a segmented scanning transmission electron microscope detector, and an integrated differential phase contrast detector.
[0097] Embodiment 12 includes the subject matter of any one of Embodiments 10-11, and further provides that the at least two different detector modalities include one or both of a diffraction detector modality and a spectral detector modality.
[0098] Embodiment 13 includes the subject matter of any one of Embodiments 1 to 12, and further provides that the trained machine learning model is trained using a mixture of supervised learning and unsupervised learning.
[0099] Embodiment 14 includes the subject matter of any one of Embodiments 1 to 13, and further provides that the trained machine learning model is trained using a recurrent generative adversarial network.
[0100] Embodiment 15 includes the subject matter of any one of Embodiments 1 to 14, and further provides that the trained machine learning model includes a convolutional neural network.
[0101] Embodiment 16 includes the subject matter of any one of Embodiments 1 to 15, and further provides that displaying the enhanced image comprises applying image segmentation to the predicted image to find the atomic positions.
[0102] Embodiment 17 includes the subject matter of embodiment 16, and further provides that applying image segmentation to the predicted image includes classifying pixels of the predicted image based on a pixel intensity threshold.
[0103] Embodiment 18 is a method for scanning a sample using a charged particle microscope system, the method comprising: adjusting the sample to different positions relative to the field of view of a charged particle beam at different times during image acquisition; scanning an area of the sample using the charged particle beam at one of these different positions and under a first imaging condition; acquiring one or more first images of the area from the scan under the first imaging condition; applying the one or more first images to the input of a trained machine learning model to obtain a predicted image indicating the probability of an atomic structure in the area of the sample; in response to obtaining the predicted image, displaying an enhanced image indicating the positions of atoms in the area of the sample based on the atomic structure probability in the predicted image; scanning the area of the sample using the charged particle beam at one of the different positions and under a second imaging condition different from the first imaging condition; and acquiring a second image of the area of the sample from the scan under the second imaging condition.
[0104] Embodiment 19 includes the subject matter of Embodiment 18, and further provides that the signal-to-noise ratio of the one or more first images is lower than the signal-to-noise ratio of the second image.
[0105] Embodiment 20 is a charged particle microscope support device, comprising: a first logic, which is used to enable the charged particle microscope system to generate one or more first images of the sample having a signal-to-noise ratio below a threshold; a second logic, which is used to apply the one or more first images to the input of a trained machine learning model to generate a predicted image indicating the probability of an atomic structure in the sample; a third logic, which is used to generate an enhanced image that reveals the positions of atoms in the sample based on the atomic structure probability in the predicted image; and a fourth logic, which is used to enable the charged particle microscope system to generate a second image of the sample having a signal-to-noise ratio above the threshold.
[0106] Embodiment 21 is a system for scanning a sample, the system comprising: a sample holder, the sample holder being configured to hold the sample; a charged particle source, the charged particle source being configured to emit a charged particle beam toward the sample; an optical system, the optical system being configured to cause the charged particle beam to be incident on the sample; one or more detectors, the one or more detectors being configured to detect charged particles of the charged particle beam and / or radiation generated by the charged particle beam incident on the sample; one or more processors; and a memory, the memory storing computer-readable instructions, which when executed by the one or more processors cause the system to: scan an area of the sample using the charged particle beam during image acquisition; acquire one or more first images of the area of the sample under first imaging conditions; apply the one or more first images to the input of a trained machine learning model to obtain a predicted image indicating a probability of an atomic structure in the area of the sample; and cause an enhanced image revealing the positions of atoms in the area of the sample to be displayed based on the atomic structure probability in the predicted image.
[0107] Embodiment 22 includes the subject matter described in embodiment 21, and further provides that the computer-readable instructions, when executed by the one or more processors, also cause the system to acquire a second image of the area of the sample under a second imaging condition, wherein the second imaging condition is selected based on the predicted image or the enhanced image so that the signal-to-noise ratio of the second image is higher than the signal-to-noise ratio of the one or more first images.
Claims
1. A method for imaging a sample using a charged particle microscope system, the method include: acquiring one or more first images of a region of the sample under first imaging conditions using the charged particle microscope system; applying the one or more first images to an input of a trained machine learning model to obtain a predicted image indicative of atomic structure probabilities in the region of the sample; as well as In response to obtaining the predicted image, an enhanced image indicating atomic positions in the region of the sample based on the atomic structure probability in the predicted image is displayed.
2. According to the method according to claim 1, the method also includes using the charged particle microscope system to collect a second image of the area of the sample under a second imaging condition, wherein the second imaging condition is selected based on the predicted image or the enhanced image so that the signal-to-noise ratio of the second image is higher than the signal-to-noise ratio of the one or more first images.
3. The method of claim 2, wherein acquiring the second image of the area of the sample is triggered from a user interface after displaying the enhanced image. 4 . The method according to claim 2 , wherein a charged particle beam dose under the first imaging condition is lower than a charged particle beam dose under the second imaging condition.
5. The method of any one of claims 2 to 4, wherein a first scanning pattern used in acquiring the one or more first images is sparse compared to a second scanning pattern used in acquiring the second image. 6 . The method according to claim 1 , further comprising sequentially displaying the one or more first images and at least one of the predicted image and the enhanced image on a user interface.
7. The method of any one of claims 1 to 6, wherein acquiring the one or more first images of the area of the sample comprises scanning a charged particle beam over the area of the sample.
8. The method of claim 7, wherein acquiring the one or more first images of the area of the sample further comprises collecting image data from the area of the sample using a single detector modality and constructing the one or more first images based on the collected image data.
9. The method of claim 8, wherein the single detector modality is a high angle annular dark field detector modality.
10. The method of claim 7, wherein acquiring the one or more first images of the area of the sample further comprises collecting image data from the area of the sample using at least two different detector modalities and constructing the one or more first images based on the collected image data.
11. A method according to claim 10, wherein the at least two different detector modalities include one or more of the following: a dark field detector modality, an annular dark field detector modality, a bright field detector modality, a high-angle annular dark field detector, a segmented scanning transmission electron microscope detector, and an integrated differential phase contrast detector.
12. The method of any one of claims 10 to 11, wherein the at least two different detector modalities include one or both of a diffraction detector modality and a spectral detector modality.
13. A method according to any one of claims 1 to 12, wherein the trained machine learning model is trained using a mixture of supervised learning and unsupervised learning.
14. The method of claim 13, wherein the trained machine learning model is trained using a recurrent generative adversarial network.
15. A method according to any one of claims 1 to 14, wherein the trained machine learning model comprises a convolutional neural network.
16. The method of any one of claims 1 to 15, wherein causing display of the enhanced image comprises applying image segmentation to the predicted image to find the atomic positions.
17. The method of claim 16, wherein applying image segmentation to the predicted image comprises classifying pixels of the predicted image based on a pixel intensity threshold.
18. A method for scanning a sample using a charged particle microscope system, the method include: adjusting the sample to different positions relative to the field of view of the charged particle beam at different times during image acquisition; scanning a region of the sample with a charged particle beam at one of the different locations and under first imaging conditions; acquiring one or more first images of the area from the scan under the first imaging condition; applying the one or more first images to an input of a trained machine learning model to obtain a predicted image indicative of atomic structure probabilities in the region of the sample; In response to obtaining the predicted image, causing an enhanced image to be displayed indicating atomic positions in the region of the sample based on the atomic structure probabilities in the predicted image; scanning the region of the sample with the charged particle beam at the one of the different locations and under a second imaging condition different from the first imaging condition; as well as A second image of the area of the sample is acquired from the scan under the second imaging condition. The method of claim 18 , wherein a signal-to-noise ratio of the one or more first images is lower than a signal-to-noise ratio of the second image.
20. A charged particle microscope support device, include: first logic for causing the charged-particle microscope system to generate one or more first images of the sample having a signal-to-noise ratio below a threshold; second logic for applying the one or more first images to an input of a trained machine learning model to generate a predicted image indicative of probabilities of atomic structures in the sample; third logic for generating an enhanced image revealing atomic locations in the sample based on the atomic structure probabilities in the predicted image; and and fourth logic for causing the charged-particle microscope system to generate a second image of the sample having a signal-to-noise ratio above the threshold.
21. A system for scanning a sample, the system include: a sample holder configured to hold the sample; a charged particle source configured to emit a charged particle beam toward the sample; an optical system configured to cause the charged particle beam to be incident on the sample; one or more detectors configured to detect charged particles of the charged particle beam and / or radiation generated by the charged particle beam incident on the sample; one or more processors; and a memory storing computer-readable instructions that, when executed by the one or more processors, cause the system to: scanning a region of the sample with the charged particle beam during image acquisition; acquiring one or more first images of a region of the sample under first imaging conditions; applying the one or more first images to an input of a trained machine learning model to obtain a predicted image indicative of atomic structure probabilities in the region of the sample; as well as An enhanced image is caused to be displayed that reveals atomic positions in the region of the sample based on the atomic structure probability in the predicted image.
22. A system according to claim 21, wherein the computer-readable instructions, when executed by the one or more processors, further cause the system to acquire a second image of the area of the sample under a second imaging condition, wherein the second imaging condition is selected based on the predicted image or the enhanced image so that the signal-to-noise ratio of the second image is higher than the signal-to-noise ratio of the one or more first images.
Citation Information
Patent Citations
Live-assisted image acquisition method and system with charged particle microscopy
US20230395351A1
