Actual situation support type image acquisition method and system using charged particle microscopy

Abstract

A method of imaging a sample includes acquiring one or more first images of a region of the sample under first imaging conditions using a charged particle microscope system. The one or more first images are applied as an input to a trained machine learning model to obtain a predicted image indicating the atomic structure probability in the region of the sample. In response to the acquisition of the predicted image, an enhanced image indicating the atomic positions within the region of the sample is displayed based on the atomic structure probability in the predicted image.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims priority to U.S. Patent Application No. 17 / 831,147, filed on June 2, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] (Technical Field) This field relates to charged - particle microscopy.

Background Art

[0003] Charged - particle microscopy involves using a beam of accelerated charged particles as an illumination source. Exemplary types of charged - particle microscopy include transmission electron microscopy, scanning electron microscopy, scanning transmission electron microscopy, and focused ion beam microscopy.

[0004] Scanning transmission electron microscopes, also known as STEM (Scanning Transmission Electron Microscope), can be used to obtain high - resolution images of samples at the atomic scale. In STEM imaging, an electron beam is scanned across a sample or a region of the sample. The electrons interact with the sample, resulting in elastically scattered electrons that exit the sample. In the transmission imaging mode, the electrons that pass through the sample are detected and used to form a microscopic image of the sample.

[0005] STEM atomic spectroscopy can provide insights into the structure of materials. However, it can be difficult to obtain atomic images with sufficient resolution for material structure analysis. In a typical atomic spectroscopy workflow, an operator navigates around a sample to find a region of interest on the sample. When the region of interest is found, a final image of the region of interest is taken. Navigation involves scanning an electron beam across the sample or a region of the sample. To obtain an atomic resolution image, the electron beam should be aligned with the atomic rows within the irradiated area of the sample, which may require tilting the sample with respect to the electron beam path to find the desired alignment. Many of the materials of interest have a limited electron dose budget. For these materials, long navigation to find the region of interest and / or navigation using a high radiation dose (e.g., 100e - / Å 2 ) can damage the sample before the final image is acquired. BRIEF DESCRIPTION OF THE DRAWINGS

[0006]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 4A

Figure 4B

Figure 5

Figure 6

Best Mode for Carrying Out the Invention

[0007] (General Considerations) The subject matter is described along with implementation forms and examples. In some cases, as will be recognized by those skilled in the art, the disclosed implementation forms and examples can be implemented without one or more of the specific details disclosed, or can be practiced using other methods, structures, and materials not specifically disclosed herein. All implementation forms and examples described herein and shown in the drawings can be combined without any limitation to form any number of combinations, such as when the proposed combinations involve elements that are incompatible or mutually exclusive, unless the context clearly dictates otherwise. The sequential order of operations in any process described herein can be rearranged unless the context clearly dictates otherwise, such as when one operation requires the result of another operation as input.

[0008] For the sake of brevity and for the continuity of the description, the same or similar reference numerals may be used for the same or similar elements in different figures, and the description of an element in one figure is considered to carry over to other figures in which that element appears with the same or similar reference numerals. In some cases, the term "corresponding to" may be used to describe the correspondence between elements in different drawings. In an exemplary use, when an element in a first figure is described as corresponding to another element in a second figure, unless otherwise stated, the element in the first figure is considered to have the characteristics of the other element in the second figure, and vice versa.

[0009] The term "comprise", as well as its derivatives such as "comprises" and "comprising", should be construed in an open and inclusive sense, i.e., "including, but not limited to". The singular forms "a", "an", "at least one" and "the" include the plural forms unless the context clearly dictates otherwise. The term "and / or" means any one or more of the listed elements when used between the last two elements of a list of elements. The term "or" is generally used in its broadest sense, i.e., to mean "and / or" unless the context clearly dictates otherwise. When used to describe a dimension range, the phrase "between X and Y" represents a range that includes X and Y. As used herein, "apparatus" can refer to any individual device, a set of devices, a part of a device, or a set of parts of a device.

[0010] Summary The subject matter disclosed herein relates to imaging of a sample by charged particle microscopy under imaging conditions that can result in an acquired image having a low signal-to-noise ratio (SNR). In these low SNR images, the atomic structure within the imaged region of the sample is not clearly visible to the naked eye due to the low resolution of the image. The methods and systems disclosed herein can use low SNR images acquired by charged particle microscopy to generate an enhanced image that shows information about the atomic structure within the imaged region of the sample. The enhanced image can be displayed on a user interface to provide a live assist to an operator while navigating the sample to find important regions of the sample (e.g., non-uniform structures) or while capturing dynamic effects in the sample composition during in vivo in-situ experiments.

[0011] In some embodiments, the methods and systems disclosed herein may be configured to obtain image data of a region of a sample, generate one or more initial images from the image data, and generate one or more enhanced images that indicate information regarding the atomic structure within the region of the sample. In various embodiments, generating the one or more enhanced images includes using a trained machine learning model that accepts the initial image(s) as input to predict the atomic structure probability in the sample. The predicted atomic structure probability may be used to detect atomic positions within the region of the sample.

[0012] In various embodiments, the imaging conditions that result in an acquired image with a low SNR may include a low dose of the charged particle beam, a short dwell time (or high scan speed) of the charged particle beam, or a sparse (sparse) scan. Thus, the methods and systems disclosed herein may enable in-situ, low charged particle beam dose and / or high speed scanning and / or sparse (sparse) scanning of the sample. In some embodiments, the method and system may enable imaging of the sample using a charged particle beam dose that is significantly less than the dose required to obtain a high-resolution image of the sample. Advantageously, the region of the sample may be repeatedly scanned at a low dose during in-situ support before the cumulative irradiation at the low dose becomes equivalent to the irradiation at a single high-dose scan. Also, the low dose may be useful to avoid scenarios where the charged particle beam changes the dynamics of the structure within the sample while capturing an image of the sample using in-situ support.

[0013] (Embodiment - In-situ Sample Imaging System) FIG. 1 shows an embodiment of a system 100 for in-situ sample imaging. The system 100 includes a charged particle microscope system 104 for investigating and analyzing a sample. The system 100 may further include a computing environment 116 having a sample imaging application 200. In the illustrated embodiment, the charged particle microscope system 104 includes a charged particle microscope 108 enclosed within a vacuum chamber 120 and a control device 112 that may be inside or outside the vacuum chamber 120. The control device 112 is communicatively coupled to the computing environment 116.

[0014] The control device 112 is connected to various components of the charged particle microscope 108, can communicate control / power signals to the components, and can receive data from the components. The control device 112 enables control of the operation of the charged particle microscope system 104 from the sample imaging application 200, and enables the image data acquired by the charged particle microscope 108 to be transmitted to the computing environment 116 and used by the sample imaging application 200. The control device 112 can be implemented in any suitable combination of hardware and software.

[0015] Any type of charged particle microscope 108 suitable for obtaining an image of a sample can be used in the system 100. For the sake of explanation, the charged particle microscope 108 is shown as a scanning transmission electron microscope (STEM), but the charged particle microscope 108 is not limited to the specific STEM configuration shown. The STEM can be operated in a STEM mode (i.e., the charged particle beam is scanned across a region of the sample) or in a transmission electron microscope (TEM) mode (i.e., the beam is not scanned). Other examples of microscopes can include, but are not limited to, cryo-electron microscopes (cryoEM), ion microscopes, and proton microscopes.

[0016] The charged particle microscope 108 includes a first electro-optical system 144 that defines an optical axis 124. The sample 128 to be investigated and / or analyzed can be positioned along the optical axis 124 under the first electro-optical system 144. The sample 128 is supported by a sample holder 132 (or stage), which can have the ability to translate, rotate, and / or tilt the sample in some embodiments. The control device 112 is connected to the sample holder 132 and can provide a sample position control signal to the sample holder 132. The sample holder 132 can enable different regions of the sample to be positioned and / or tilted with respect to the optical axis 124 (e.g., during navigation on the sample to find a region of interest on the sample).

[0017] The charged particle microscope 108 includes a charged particle source 136 positioned above the first electron optical system 144. The charged particle source 136 can be, for example, an electron source (e.g., a Schottky gun), a positive ion source (e.g., a gallium ion source or a helium ion source), a negative ion source, a proton source, or a positron source. The charged particle source 136 generates a charged particle beam 140. The first electron optical system 144 receives the charged particle beam 140 and configures the charged particle beam 140 within the field of view on the sample 128.

[0018] The first electro-optical system 144 can include one or more electro-optical components. In some embodiments, the electro-optical components can be connected to the control device 112 such that the electro-optical power (refractive power, optical power) of the components can be set and adjusted by the control device 112. By way of example, the first electro-optical system 144 can include condenser lenses 148a, 148b, a condenser aberration corrector 148c, and a condenser aperture 148d. The first electro-optical system 144 can include a scanning coil 150 that can be operated to scan the charged particle beam 140 across the region of the sample 128. In other embodiments, the first electro-optical system 144 can have other components or fewer components than those illustrated.

[0019] When the charged particle beam 140 is incident on the sample 128, the charged particles in the beam interact with the structures (e.g., atoms) in the sample 128, and various types of radiation are emitted from the sample 128. For example, when the charged particle beam 140 is an electron beam, the radiation emitted from the sample can include any combination of Auger electrons, secondary electrons, X-rays, backscattered electrons, cathode ray luminescence, loss electrons, transmitted electrons, and diffracted electrons. Various types of radiation can be detected and used to form one or more images of the sample.

[0020] In some embodiments, the charged particle microscope 108 may include one or more detector systems for capturing an image of the sample 128 using one or more detector modalities. In one embodiment, the charged particle microscope 108 may include a first detector system 152 for acquiring image data from the sample 128. The first detector system 152 may include a bright field detector, an annular bright field detector, a dark field detector, an annular dark field detector, a high-angle annular dark field (HAADF) detector, a segmented STEM detector, or a STEM detector such as an integrated differential phase contrast (iDPC) detector. In one particular embodiment, the first detector system 152 may include a HAADF detector that detects charged particles (e.g., electrons) scattered at high angles. In another particular embodiment, the first detector system 152 may include an annular bright field detector and an annular dark field detector for simultaneously capturing bright field and dark field images of the sample.

[0021] In one embodiment, the charged particle microscope 108 may include one or more additional detector systems, such as a second detector system 156 and a third detector system 160, for acquiring additional image data from the sample. For example, the detector systems 156, 160 may be spectroscopic systems. For illustrative purposes, the second detector system 156 is shown positioned below the sample 128 and the third detector system 160 is shown positioned above the sample 128. In one particular embodiment, the second detector system 156 may include electron energy loss spectroscopy and the third detector system 160 may include energy dispersive X-ray spectroscopy.

[0022] The charged particle microscope 108 may include a second optical system 164 that directs charged particles transmitted through the sample 128 into the fields of view of the detector systems 152, 156. For purposes of illustration, the second optical system 164 may include an objective lens 168a, an objective aberration corrector 168b, and an objective aperture 168c. In other embodiments, the second optical system 164 may have other components or fewer components than those shown.

[0023] The computing environment 116 may have any suitable configuration for executing the sample imaging application 200. For example, the computing environment 116 may include a processing device 180, a memory 184, a display device 188, and a data storage device 192. The sample imaging application 200 may be loaded into the memory 184 and executed by the processing device 180. The sample imaging application 200 may present a user interface on the display device 188, present a sample image within the user interface, and adjust microscope control settings from the user interface. The sample imaging application 200 may provide microscope control settings to the control device 112 and receive detector data from the control device 112. The computing environment 116 may include other components not specifically shown, such as input device(s), other output device(s) other than the display device 188, communication connection(s), other memory other than the memory 184, and other processing device(s) other than the processing device 180.

[0024] (Example - Sample Imaging Application) Figure 2 shows an example of a sample imaging application 200. The sample imaging application 200 may include imaging logic 204, atomic structure prediction logic 208, atomic position detection logic 210, and user interface logic 216. The imaging logic 204 may perform various functions related to acquiring an image of a sample. The atomic structure prediction logic 208 may perform various functions related to predicting the atomic structure probability in the acquired image of the sample. The atomic position detection logic 210 may perform various functions related to finding the atomic positions in a prediction image including the atomic structure probability. The user interface logic 216 may perform various functions related to presenting the acquired and highlighted image of the sample to the user and receiving input from the user. The sample imaging application 200 may have other logical components and data structures not specifically illustrated.

[0025] The imaging logic 204 may perform various functions related to acquiring an image of a sample using a charged particle microscope system. During navigation on the sample, the imaging logic 204 may position the sample at different positions relative to the field of view of the charged particle beam. For example, the imaging logic 204 may provide position information regarding the sample to a control device of the charged particle microscope system (e.g., the control device 112 of FIG. 1). In some embodiments, the imaging logic 204 may derive the position information of the sample from user-defined settings received from the user interface logic 216. The control device may then provide a control set point to a sample holder (e.g., the sample holder 132 of FIG. 1) to translate and / or tilt the sample. In some cases, when a charged particle microscope (e.g., the charged particle microscope 108 of FIG. 1) is mounted on a stage, the charged particle microscope may be translated and / or tilted relative to the sample to achieve a desired relative positioning between the sample and the scanning field of the charged particle beam.

[0026] The imaging logic 204 can cause different types of images of a sample to be captured by a charged particle microscope system. For example, the imaging logic 204 can provide a detector system to be activated during imaging of a region of the sample to a control device of the charged particle microscope system. In one embodiment, the imaging logic 204 can use one detector modality (e.g., HAADF detector modality) or at least two detector modalities (e.g., bright field detector modality and dark field detector modality, or HAADF detector modality and spectroscopic detector modality) to obtain an image of the sample.

[0027] The imaging logic 204 can receive detector data from the charged particle microscope system. In some embodiments, the imaging logic 204 can construct an initial image of the sample from the detector data. For example, when a charged particle beam scans across a region of the sample (e.g., in a raster pattern), the charged particle microscope detector system generates an output for each (x, y) scan beam position and tilt angle of the sample. The detector output for each scan beam position can provide information for a pixel of the sample image. The imaging logic 204 can use the detector output to construct one or more initial images of the sample. In some embodiments, the imaging logic 204 can apply a time stamp to the constructed initial image and store the initial image in a data storage device (e.g., data storage device 192 of FIG. 1) for use by other components of the sample imaging application 200.

[0028] The atomic structure prediction logic 208 can perform various functions related to predicting the atomic structure probability in the initial image generated by the imaging logic 204. In some embodiments, the initial image may have a low SNR. FIG. 3A shows an example of a low SNR image 300 obtained from a region of a sample by a HAADF detector. In some embodiments, the atomic structure prediction logic 208 can obtain one or more low SNR images, such as the low SNR image 300 shown in FIG. 3A, and predict the atomic structure probability in the region of the sample corresponding to the one or more low SNR images. The output of the atomic structure prediction logic 208 can be an atomic structure probability image, which is an image in which a detectable contrast exists between regions of the image that are likely to contain an atomic structure and regions of the image that are likely not to contain an atomic structure.

[0029] In some embodiments, the atomic structure prediction logic 208 uses a trained machine learning model to predict the atomic structure probability. For example, the trained machine learning model can be a neural network (e.g., a convolutional neural network) trained to predict the atomic structure probability. In some embodiments, the atomic structure prediction logic 208 can generate an inference request that includes a set of one or more initial images (or acquired images). The set of one or more initial images can form an input vector for the trained neural network. In some embodiments, the set of one or more initial images can be images acquired with the same detector modality. In other embodiments, the set of one or more initial images can be images acquired using different detector modalities. The atomic structure prediction logic 208 can send the inference request to an inference engine 220 that includes a trained machine learning model 224. The inference engine 220 can be in the same computing environment as the sample imaging application 200 or in a different computing environment (e.g., on an AI server in the cloud).

[0030] When the inference request is received, the inference engine 220 applies a set of one or more initial images within the inference request to the input of the trained machine learning model 224 in order to obtain a prediction. The prediction can be an image that includes atomic structure probabilities. FIG. 3B shows an example of a predicted image 310 output by a neural network trained based on the low SNR image 310 shown in FIG. 3A (the estimated region of the image 300 including the atomic structure has a higher pixel intensity compared to the region of the image with low likelihood of including the atomic structure). The inference engine 220 generates an inference that includes the predicted image. The inference engine 220 may add other information, such as the confidence of the neural network's prediction and / or an explanation of the neural network's prediction, to the inference. The inference engine 220 returns the inference to the atomic structure prediction logic 208.

[0031] The confidence of the prediction made by the trained machine learning model 224 can be determined using the same techniques that exist for the validation of artificial neural networks. For example, the trained machine learning model 224 can be applied to low-quality images (or low SNR images), and the predictions of the trained machine learning model 224 can be compared to high-quality images (or high SNR images) in order to determine the confidence of the predictions. The low-quality images and high-quality images can either be generated under different imaging conditions or can be simulated, either by being obtained from either measurements or simulations and artificially degrading the high-quality images obtained.

[0032] The atomic position detection logic 210 can extract atomic positions from a predicted image including atomic structure probabilities. In some embodiments, the atomic position detection logic 210 uses image segmentation to extract atomic positions from the atomic structure probability image. Various types of image segmentation techniques can be used. One technique can be based on thresholding. In thresholding, a pixel intensity threshold is set to classify the pixels in the atomic structure probability image into atomic structure pixels and background pixels. The atomic position detection logic 210 can output the atomic positions or can output an enhanced image indicating the atomic positions. In some embodiments, the enhanced image can be generated by overlaying the atomic positions on the predicted image. FIG. 3C shows an example of an enhanced image 320 that includes atomic positions determined by the atomic position detection logic 210 overlaid on the predicted image 310 of FIG. 3B (the large dots indicate the atomic positions).

[0033] The user interface logic 216 can perform various functions related to displaying a user interface on a display device (e.g., the display device 188 of FIG. 1), responding to events triggered from the user interface, and receiving user input from the user interface. For example, the user interface logic 216 can receive images from the atomic structure prediction logic 208 and / or the atomic position detection logic 210. The user interface logic 216 can cause a graphical display of at least a portion of the image to be presented in a specified region of the user interface. In one embodiment, during navigation on a sample or in-situ experimentation, when a region of the sample is scanned, the user interface logic 216 can present a series of images within the user interface. The series of images can include one or more acquired images of the region of the sample and one or more enhanced images generated based on the acquired image(s).

[0034] The user interface logic 216 can receive inputs entered by a user in the user interface. In one embodiment, the input can be a setting of the charged particle microscope system or an annotation to an image presented in the user interface. In some cases, the user can select a control on the user interface to initiate imaging of a region of a sample. In this case, the user interface logic 216 can receive a request to initiate imaging and transmit that request to the imaging logic 204. The user interface logic 216 can cause other information, such as one or more metrics related to the operation of a trained machine learning model within the inference engine 220, to be displayed within the user interface.

[0035] In some cases, the sample imaging application 200 can include training logic 214 that can generate a request to train or retrain a machine learning model to predict atomic structure probabilities from low SNR images. In one embodiment, the training logic 214 can generate a training request for the training of the machine learning model 228 and transmit that training request to the training engine 230.

[0036] The training engine 230 can receive a training request from the training logic 214 and perform a task of training the machine learning model 228 to predict atomic structure probabilities in low SNR images. In some embodiments, the training engine 230 can be in the same computing environment as the sample imaging application 200. In other embodiments, the training engine 230 can be in a remote machine learning environment within the cloud. After training the machine learning model 228, the training engine 230 can notify the training logic 214. In response to the notification, the training logic 214 can deploy the trained machine learning model 224 (generated by the training of the machine learning model 228) to the inference engine 220 for use in making predictions.

[0037] The sample imaging application 200 can be stored in one or more computer-readable storage media or computer-readable storage devices and executed by one or more processing devices (e.g., the processing device 180 within the computing environment 116 shown in FIG. 1). Any of the computer-readable media herein can be non-transitory (e.g., volatile memory such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory), magnetic storage devices, optical storage devices, etc. non-volatile memory) and / or can be tangible.

[0038] (Example - Training Dataset for Machine Learning) The training engine 230 can receive a training request from the sample imaging application 200. The training request may include a training dataset 232, or the training engine 230 can retrieve the training dataset 232 from a data storage device (e.g., the data storage device 192 in FIG. 1), or the training engine 230 can generate the training dataset 232. The training dataset 232 can include any combination of simulated images of atomic structures, actual images, and acquired images. Preferably, the training dataset 232 includes images from conditions similar to the target use case and materials (e.g., noise statistics, as well as level grid structure and atomic grid structure).

[0039] In one embodiment, the dataset 232 may include low SNR images of a sample obtained using a charged particle microscope system. Each low SNR image may be tagged with the name of the sample and the region on the sample where the low SNR image was obtained. The training dataset may further include high SNR images of the sample obtained using a charged particle microscope system. Each high SNR image may be tagged with the name of the sample and the region on the sample where the high SNR image was obtained. Input-output pairs for the training dataset 232 may be generated by pairing each low SNR image (as an input) with one of the high SNR images (as an output) using matching sample names and region tags as the pairing criterion. In some cases, two or more low SNR images obtained using different detector modalities (as inputs) may be paired with one of the high SNR images (as an output). In some cases, instead of directly obtaining low SNR images from the charged particle microscope system, the obtained high SNR images may be artificially degraded to form the low SNR images used in the training dataset.

[0040] In another embodiment, the training dataset 232 may include simulated images of atomic structures generated based on an atomic structure model and a noise model. The machine learning model may be trained and executed to generate the simulated images. One example of a technique for generating the simulated images is described in Lin, R., Zhang, R., Wang, C., et al., TEM ImageNet training library and AtomSegNet deep - learning models for high - precision atom segmentation, localization, denoising, and deblurring of atomic - resolution images. Sci Rep 11, 5386 (2021).

[0041] In another example, the training dataset may include real images obtained from open access electron microscopy datasets, such as the Warwick Electron Microscopy Dataset, which is available at github.com / Jeffrey-Ede / datasets / wiki and described in "Warwick Electron Microscopy Datasets" by Jeffrey M Ede, March 2020, Learn. Sci. Technol. 1 045003.

[0042] (Example - Training of Machine Learning Model) In one example, the training engine 230 trains the machine learning model 228 using teacher - free training cycles interleaved with teacher - assisted training cycles. In one example, after several teacher - assisted cycles (e.g., two or more teacher - assisted cycles), several teacher - free cycles (e.g., two or more teacher - free cycles) can follow.

[0043] In one example, teacher - assisted training and teacher - free training may use a Cycle - Generative Adversarial Network (CycleGAN) model architecture (see "Unpaired Image - to - Image Translation using Cycle - Consistent Adversarial Networks" by Jun - Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros, arXiv:1703.10593v7, August 24, 2020).

[0044] Figure 6 shows a CycleGAN architecture including two generators G and generator F. Generator G maps data from target domain X to Y. Generator F maps data from target domain Y to X. The CycleGan architecture includes two discriminators Dx and Dy. Discriminator Dy facilitates the conversion of the output of generator G into an output where X cannot be distinguished from the target domain Y. Discriminator Dx facilitates the conversion of the output of generator F into an output where Y cannot be distinguished from the target domain X. Discriminators Dx and Dy are only used during unsupervised training.

[0045] Training includes a forward cycle and a reverse cycle. In the forward cycle, generator G receives input x and generates output Y°, and generator F receives Y° as input and generates output x°. Input x and output x° should be similar. The difference between x and x° is the cycle consistency loss in the forward direction and can be included in the optimization of the network. In the reverse cycle, generator G receives input y and generates output X°, and generator G receives X° as input and generates output y°. Input y and output y° should be similar. The difference between y and y° is the cycle consistency loss in the reverse direction and can be included in the optimization of the network.

[0046] Discriminators Dx and Dy are used for unsupervised training. In the forward cycle, discriminator Dy receives Y° and Y as input and generates an output indicating true or false. In the reverse cycle, discriminator Dx receives X° and X as input and generates an output indicating true or false. In unsupervised training, the input images x and y are not paired and are noisy images (e.g., low SNR images). Images X and Y are real images (e.g., high SNR images).

[0047] During teacher-aided training, discriminators Dx and Dy are not used. The G network is executed independently of the large structure using the simulated image as input x and the real image paired as label y. The F network is also executed independently of the large structure using the real image as input y and the simulated image paired as label x.

[0048] CycleGAN alternates between these modes after several cycles in one of these modes, between teacher-aided training mode and teacher-free training mode.

[0049] The generator and discriminator can be convolutional neural networks. In one embodiment, a U-Net architecture can be used for generators G and F. U-Net is a convolutional neural network developed for biomedical image segmentation (see "U-Net Convolutional Networks for Biomedical Image Segmentation" by Olaf Ronneberger et al., arXiv:1505.04597, May 18, 2015). In one particular embodiment, a parameterized U-Net structure with an input of 256×256 having eight multi-resolution layers can be used for the generator. Other types of convolutional neural networks such as the VGG architecture or SUNet (Swin Transformer UNet for image noise removal) can be used for the generator. In one embodiment, the discriminator can be a PatchGAN discriminator, which is a convolutional network where the input image is mapped to an N×N array instead of a single scalar vector.

[0050] (Embodiment - Situation-Assisted Sample Imaging Method) FIG. 4A is a flowchart showing an exemplary method 400 for imaging a sample using a charged particle microscope system, according to one embodiment. FIG. 4B is a block diagram showing a part of method 400. Method 400 can be executed using the system shown in FIG. 1 and / or the sample imaging application shown in FIG. 2. The operations are illustrated in a specific order in FIGS. 4A and 4B, once each, respectively, but the operations may be reordered and / or repeated as desired and appropriate (e.g., different operations shown to be executed sequentially may be executed in parallel as suitable).

[0051] At 410, the method includes navigating to a region of interest (ROI) on the sample. For example, the sample can be positioned such that the ROI is within the field of view of the charged particle beam output by the charged particle microscope system. Positioning the sample can include transmitting suitable control to the sample holder (e.g., by a control device of the charged particle microscope system) to adjust the position of the sample relative to the charged particle beam.

[0052] At 420, the method includes setting imaging conditions for the sample. The imaging conditions can be set by adjusting one or more imaging parameters of the charged particle microscope system. For example, the method can include setting the dose of the charged particle beam irradiating the sample. When an image is generated by scanning the ROI, the method can include setting the dwell time or scan speed or scan pattern of the charged particle beam. The method can further include operating the detector system for use in acquiring image data. The method can further include adjusting the tilt angle of the sample. The parameter settings can originate from the sample imaging application and can be applied by a control device of the charged particle microscope system.

[0053] At 430, the method includes obtaining image data from the ROI under the imaging conditions set in operation 420. For example, while a charged particle beam is incident on the ROI, an activated detector system(s) may collect image data generated from the interaction between the charged particle beam and the ROI. In some embodiments, the ROI may be scanned by sweeping a charged particle beam across the ROI in a raster pattern (or another scanning pattern). The activated detector system(s) may capture the image data when the ROI is scanned. In some embodiments, the image data may be captured using a single detector modality (e.g., HAADF detector modality). In other embodiments, the image data may be captured using at least two different detector modalities (e.g., dark field detector modality and bright field detector modality, or diffraction detector modality (e.g., HAADF detector modality), and spectral detector modality (e.g., electron energy loss spectroscopy or energy dispersive X-ray spectroscopy)). In other embodiments, a camera may capture an image of the sample. The camera may include, for example, a CCD (Charged-Coupled Device) imaging sensor, a CMOS (Complementary Metal-Oxide-Semiconductor) imaging sensor, or more generally, an array of photodetectors. The camera may operate in “movie” mode and capture consecutive images of the sample.

[0054] At 440, the method may generate a set of one or more initial images (or acquired images) from the image data obtained in operation 430. For example, the image data may include the charged particle intensity measured at each scanning position of the charged particle beam. Each intensity may correspond to a pixel of the initial image. The initial image may be constructed as a set of pixels where the pixel coordinates are correlated with the scanning positions of the beam positions and the pixel values are determined by the measured intensities corresponding to the scanning positions of the beam. In some embodiments, the imaging conditions set in operation 420 result in initial image(s) with a low SNR (i.e., low SNR image(s)).

[0055] At 450, the method includes generating a predicted image indicative of atomic structure probability in the ROI using a trained machine learning model. In some embodiments, the trained machine learning model is a neural network (e.g., a convolutional neural network) trained to predict atomic structure probability based on one or more low SNR images. The output of the neural network can be a predicted image indicative of atomic structure probability. In some embodiments, one or more of the initial images generated at operation 440 can be applied to the input of the trained machine learning model to obtain the predicted image. For illustrative purposes, FIG. 4B shows a low SNR image 460 applied to the input of the trained machine learning model 224, and then the trained machine learning model generates a predicted image 464 indicative of atomic structure probability. Optionally, two or more low SNR images (e.g., two or more low SNR images captured with different detector modalities) can be applied to the input of the trained machine learning model 224, and the trained machine learning model 224 can output a predicted image based on the two or more low SNR images.

[0056] At 470, the method includes identifying atomic positions within the predicted image generated at operation 450. In one embodiment, image segmentation can be used to identify the atomic positions. Various types of image segmentation can be used. One example of an image segmentation technique is thresholding. In thresholding, a pixel intensity threshold is set to classify pixels within the predicted image into atomic pixels and background pixels. The method can include generating an enhanced image by overlaying an atomic object (e.g., a geometric shape representing an atom) on the predicted image at the atomic positions found by the image segmentation. For illustration, FIG. 4B shows the predicted image 464 input to the atomic position detection logic 210. FIG. 4B shows an enhanced image 468 revealing the atomic positions (large dots 469) within the predicted image 464.

[0057] At 480, the method may include presenting a live image of the sample on a display. In some embodiments, the method may include recording the image for playback after image acquisition. Any combination of the images generated at operations 440, 450, and 470 may be presented and / or recorded. FIG. 4B shows that any of images 460, 464, and 468 may be provided to the user interface logic 216, which may update the user interface live with the image. In one embodiment, an initial image (e.g., a low SNR image) may be presented, then a predicted image generated based on the initial image may be presented, and then an enhanced image generated based on the predicted image may be presented. In some embodiments, the predicted image may be displayed with a confidence level of the prediction. The image may be presented within the user interface to enable user interaction with the image. For example, the user may zoom in on a portion of the enhanced image and select or tag a portion of the image for further investigation. In another embodiment, the user may detect that the charged particle beam is not accurately aligned with the atomic column within the ROI and may determine an adjustment to be made to the tilt of the sample based on the image.

[0058] At 485, the method may include determining whether to re-acquire image data from the ROI. For example, the method may determine whether to re-acquire image data from the ROI based on user input. For example, if the user detects that the sample is not properly aligned with the charged particle beam, the user may adjust the setting of the sample tilt angle via the user interface and trigger re-acquisition of the image data of the ROI. In another embodiment, if the user detects that either the charged particle beam dose is too low or too high, the user may adjust the imaging conditions and trigger re-acquisition of the image data of the ROI. If the method determines that the image data should be re-acquired, the method may return to operation 420 and re-acquire the image data by adjusting the imaging conditions (e.g., based on user input). If the method determines that it is not necessary to re-acquire the image data of the ROI, the method may proceed to operation 490.

[0059] At 490, the method includes obtaining a final image of the ROI. The final image can be obtained under imaging conditions different from those used in operation 420 to obtain the initial image(s) of the ROI. In particular, the imaging conditions used to obtain the final image can result in a final image having a higher SNR than the initial image(s). In some embodiments, the charged particle beam dose used to obtain the final image can be more than the charged particle beam dose used to obtain the initial image(s), and / or the scanning speed used to obtain the final image can be slower than the scanning speed used to obtain the initial image(s), and / or the scanning pattern used to obtain a low SNR image can be sparser than the scanning pattern used to obtain the final image.

[0060] At 495, the method can include determining whether to navigate to another ROI on the sample. If the method determines that another ROI on the sample should be processed, the method can return to operation 410 and navigate to the new ROI. In some cases, the user can indicate through the user interface that another ROI should be processed. As part of indicating that another ROI should be processed, the user may specify a new ROI. Alternatively, the method may automatically determine a new ROI to be processed based on a predetermined experiment or navigation plan.

[0061] At 499, if the method determines that the processing of another ROI is not required, the image acquisition can be terminated.

[0062] (Example - Graphical User Interface) FIG. 5 shows an exemplary Graphical User Interface (GUI) 500 that can be presented on a display device by a sample imaging application. A user may interact with the GUI 500 using any suitable input device and input technology (e.g., cursor movement, motion capture, face recognition, gesture detection, voice recognition, button activation, etc.).

[0063] The GUI 500 may include a data display area 502, a data analysis area 504, a charged particle microscope control area 506, and a settings area 508. The specific number and arrangement of the areas shown in FIG. 5 are merely exemplary, and the GUI 500 may include any number and arrangement of areas including any desired features.

[0064] The data display area 502 may display an image generated by a sample imaging application. For example, the data display area 502 may display any of a low-resolution structure image, a structure position image, a structure object image, and a structure type image.

[0065] The data analysis area 504 may display the results of data analysis. For example, the data analysis area 504 may display a region of interest indicated by the user in the image displayed in the data display area. In some embodiments, the data display area 502 and the data analysis area 504 may be combined within the GUI 500.

[0066] The charged particle microscope control area 506 may include options that enable a user to control a charged particle microscope system. For example, the charged particle microscope control area 506 may include user-selectable options that enable adjustment of parameters of the charged particle microscope system.

[0067] The setting area 508 may include options that enable a user to control the features and functions of the GUI 500 and / or perform other computing operations with respect to the data display area 502 and the data analysis area 504 (e.g., save data to a storage device).

[0068] In view of the above implementations and examples of the disclosed subject matter, this application discloses additional examples listed below. Note that one feature of a single item, or a combination of two or more features of that item, may also be, if desired, a further example included in the disclosure of this application in combination with one or more features of one or more further examples.

[0069] Example 1 is a method of imaging a sample using a charged particle microscope system, the method comprising: using the charged particle microscope system to acquire one or more first images of a region of the sample under a first imaging condition; applying the one or more first images to an input of a trained machine learning model to obtain a predicted image indicating an atomic structure probability in the region of the sample; and in response to obtaining the predicted image, displaying an enhanced image indicating atomic positions within the region of the sample based on the atomic structure probability in the predicted image.

[0070] Example 2 includes the subject matter of Example 1 and further comprises using the charged particle microscope system to acquire a second image of a region of the sample under a second imaging condition, wherein the second imaging condition is selected based on the predicted image or the enhanced image such that the second image has a higher signal-to-noise ratio compared to the one or more first images.

[0071] Example 3 includes the subject matter of Example 2 and further specifies that acquiring the second image of the region of the sample is triggered from a user interface after the enhanced image is displayed.

[0072] Example 4 includes the subject matter described in any one of Examples 2 or 3, and further specifies that the dose of the charged particle beam under the first imaging condition is less than the dose of the charged particle beam under the second imaging condition.

[0073] Example 5 includes the subject matter described in any one of Examples 2 to 4, and further specifies that the first scanning pattern used when acquiring the one or more first images is sparser (more sparse) compared to the second scanning pattern used when acquiring the second image.

[0074] Example 6 includes the subject matter described in any one of Examples 1 to 5, and further includes sequentially displaying the one or more first images and at least one of the predicted image and the enhanced image on a user interface.

[0075] Example 7 includes the subject matter described in any one of Examples 1 to 6, and further specifies that acquiring the one or more first images of the region of the sample includes scanning a charged particle beam across the region of the sample.

[0076] Example 8 includes the subject matter of Example 7, and further specifies that acquiring the one or more first images of the region of the sample further includes collecting image data from the region of the sample using a single detector modality and constructing the one or more first images from the collected image data.

[0077] Example 9 includes the subject matter of Example 8, and further specifies that the single detector modality is a high-angle annular dark-field detector modality.

[0078] Example 10 includes the subject matter of Example 7, and further specifies that acquiring the one or more first images of the region of the sample includes collecting image data from the region of the sample using at least two different detector modalities and constructing the one or more first images from the collected image data.

[0079] Example 11 includes the subject matter of Example 10 and further specifies that the at least two different detector modalities include one or more of a dark field detector modality, an annular dark field detector modality, a bright field detector modality, a high angle annular dark field detector, a segmented scanning transmission electron microscope detector, and an integrated differential phase contrast detector.

[0080] Example 12 includes the subject matter of any one of Examples 10 to 11 and further specifies that the at least two different detector modalities include one or both of a diffraction detector modality and a spectrum detector modality.

[0081] Example 13 includes the subject matter of any one of Examples 1 to 12 and further specifies that the trained machine learning model is trained using a mixture of supervised learning and unsupervised learning.

[0082] Example 14 includes the subject matter of any one of Examples 1 to 13 and further specifies that the trained machine learning model is trained using a cycle generative adversarial network.

[0083] Example 15 includes the subject matter of any one of Examples 1 to 14 and further specifies that the trained machine learning model includes a convolutional neural network.

[0084] Example 16 includes the subject matter of any one of Examples 1 to 15 and further specifies that displaying the enhanced image includes applying image segmentation to the predicted image to find the atomic positions.

[0085] Example 17 includes the subject matter of Example 16 and further specifies that applying image segmentation to the predicted image includes classifying the pixels of the predicted image based on a pixel intensity threshold.

[0086] Example 18 is a method of scanning a sample using a charged particle microscope system, the method comprising: adjusting the sample to different positions with respect to the field of view of the charged particle beam at different times during image acquisition; scanning a region of the sample with the charged particle beam under a first imaging condition at one of the different positions; acquiring one or more first images of the region from the scanning under the first imaging condition; applying the one or more first images to an input of a trained machine learning model to obtain a prediction image indicating an atomic structure probability in the region of the sample; in response to obtaining the prediction image, displaying an emphasis image indicating atomic positions in the region of the sample based on the atomic structure probability in the prediction image; scanning the region of the sample with the charged particle beam under a second imaging condition different from the first imaging condition at the one of the different positions; and acquiring a second image of the region of the sample from the scanning under the second imaging condition.

[0087] Example 19 includes the subject matter of Example 18 and further specifies that the one or more first images have a lower signal-to-noise ratio compared to the second image.

[0088] Example 20 is a charged particle microscope support device, comprising: a first logic for causing a charged particle microscope system to generate one or more first images of a sample having a signal-to-noise ratio below a threshold; a second logic for applying the one or more first images to an input of a trained machine learning model to generate a prediction image indicating an atomic structure probability in the sample; a third logic for generating an emphasis image revealing atomic positions in the sample based on the atomic structure probability in the prediction image; and a fourth logic for causing the charged particle microscope system to generate a second image of the sample having a signal-to-noise ratio above the threshold.

[0089] Example 21 is a system for scanning a sample, comprising a sample holder configured to hold the sample, a charged particle source configured to emit a beam of charged particles towards the sample, an optical system configured to direct the beam of charged particles onto the sample, one or more detectors configured to detect charged particles of the charged particle beam and / or radiation resulting from the incidence of the charged particle beam on the sample, one or more processing devices, and a memory storing computer-readable instructions which, when executed by the one or more processing devices, cause the system to scan a region of the sample using the charged particle beam during image acquisition, acquire one or more first images of the region of the sample under a first imaging condition, apply the one or more first images to an input of a trained machine learning model to obtain a prediction image indicating the atomic structure probability in the region of the sample, and display an enhanced image revealing the atomic positions in the region of the sample based on the atomic structure probability in the prediction image.

[0090] Example 22 includes the subject matter of Example 21, further specifying that the computer-readable instructions, when executed by the one or more processing devices, cause the system to further acquire a second image of the region of the sample under a second imaging condition, the second imaging condition being selected based on the prediction image or the enhanced image such that the second image has a higher signal-to-noise ratio compared to the one or more first images.

Claims

1. A method for imaging a sample using a charged particle microscope system, comprising: obtaining, using the charged particle microscope system, one or more first images of a region of the sample under first imaging conditions; applying the one or more first images to an input of a trained machine learning model to obtain a prediction image indicating the atomic structure probability in the region of the sample; in response to obtaining the prediction image, displaying an enhanced image indicating atomic positions within the region of the sample based on the atomic structure probability in the prediction image. A method.

2. The method according to claim 1, further comprising obtaining a second image of the region of the sample under second imaging conditions using the charged particle microscope system, wherein the second imaging conditions are selected based on the prediction image or the enhanced image such that the second image has a higher signal-to-noise ratio compared to the one or more first images.

3. The method according to claim 2, wherein obtaining the second image of the region of the sample is triggered from a user interface after the enhanced image is displayed.

4. The method according to claim 2, wherein the charged particle beam dose in the first imaging conditions is less than the charged particle beam dose in the second imaging conditions.

5. The method according to claim 2, wherein a first scanning pattern used when obtaining the one or more first images is sparser compared to a second scanning pattern used when obtaining the second image.

6. The method according to claim 1, further comprising sequentially displaying the one or more first images and at least one of the prediction image and the enhanced image on a user interface.

7. The method according to claim 1, wherein obtaining the one or more first images of the region of the sample includes scanning a charged particle beam over the region of the sample. **Claim 8** The method according to claim 7, wherein obtaining the one or more first images of the region of the sample further includes collecting image data from the region of the sample using a single detector modality and constructing the one or more first images from the collected image data. **Claim 9** The method according to claim 8, wherein the single detector modality is a high-angle annular dark-field detector modality. **Claim 10** The method according to claim 7, wherein obtaining the one or more first images of the region of the sample further includes collecting image data from the region of the sample using at least two different detector modalities and constructing the one or more first images from the collected image data. **Claim 11** The method according to claim 10, wherein the at least two different detector modalities include one or more of a dark-field detector modality, an annular dark-field detector modality, a bright-field detector modality, a high-angle annular dark-field detector, a segmented scanning transmission electron microscope detector, and an integrated differential phase contrast detector. **Claim 12** The method according to claim 10, wherein the at least two different detector modalities include one or both of a diffraction detector modality and a spectrum detector modality. **Claim 13** The method according to claim 1, wherein the trained machine learning model is trained using a mixture of supervised learning and unsupervised learning. **Claim 14** The method according to claim 13, wherein the trained machine learning model is trained using a cycle-generative adversarial network. **Claim 15** The method according to claim 1, wherein the trained machine learning model includes a convolutional neural network.

16. The method according to claim 1, wherein displaying the highlighted image includes applying image segmentation to the predicted image to find the atomic positions.

17. The method according to claim 16, wherein applying image segmentation to the predicted image includes classifying pixels of the predicted image based on a pixel intensity threshold.

18. A method for scanning a sample using a charged particle microscope system, comprising: adjusting the sample to different positions with respect to the field of view of the charged particle beam at different times during image acquisition; scanning a region of the sample with a charged particle beam under a first imaging condition at one of the different positions; acquiring one or more first images of the region from the scanning under the first imaging condition; applying the one or more first images to an input of a trained machine learning model to obtain a predicted image indicating the atomic structure probability in the region of the sample; in response to obtaining the predicted image, displaying a highlighted image indicating atomic positions within the region of the sample based on the atomic structure probability in the predicted image; scanning the region of the sample with the charged particle beam under a second imaging condition different from the first imaging condition at the one of the different positions; acquiring a second image of the region of the sample from the scanning under the second imaging condition. Method.

19. The method according to claim 18, wherein the one or more first images have a lower signal-to-noise ratio compared to the second image.

20. A charged particle microscope support device, comprising: A first logic that causes a charged particle microscope system to generate one or more first images of a sample having a signal-to-noise ratio below a threshold; A second logic that applies the one or more first images to an input of a trained machine learning model to generate a prediction image indicating an atomic structure probability in the sample; A third logic that generates an enhanced image revealing atomic positions in the sample based on the atomic structure probability in the prediction image; A fourth logic that causes the charged particle microscope system to generate a second image of the sample having a signal-to-noise ratio above the threshold, and a charged particle microscope support device. A charged particle microscope support device.

21. A system for scanning a sample, comprising: A sample holder configured to hold the sample; A charged particle source configured to emit a charged particle beam toward the sample; An optical system configured to direct the charged particle beam onto the sample; One or more detectors configured to detect charged particles of the charged particle beam and / or radiation resulting from the incidence of the charged particle beam on the sample; One or more processing devices; A memory storing computer-readable instructions that, when executed by the one or more processing devices, cause the system to: During image acquisition, scan a region of the sample using the charged particle beam; Acquire one or more first images of the region of the sample under a first imaging condition; Apply the one or more first images to an input of a trained machine learning model to obtain a prediction image indicating an atomic structure probability in the region of the sample; A memory storing computer-readable instructions that cause an enhanced image revealing atomic positions in the region of the sample to be displayed based on the atomic structure probability in the prediction image. A system.

22. When the computer-readable instructions are executed by the one or more processing devices, the system is further caused to acquire a second image of the region of the sample under a second imaging condition, the second imaging condition being selected based on the predicted image or the enhanced image such that the second image has a higher signal-to-noise ratio compared to the one or more first images. The system according to claim 21.

Images