Systems, Apparatus, and Methods for Determining Elemental Composition Using 4D STEM - Patent application

The use of a monolithic active pixel array sensor with a global shutter readout in 4D STEM addresses the limitations of asynchronous readout and slow frame rates, enabling high-resolution, efficient elemental identification in thick biological samples.

JP7577341B2Active Publication Date: 2024-11-05DIRECT ELECTRON LP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2021560342
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-04-19
Filing Date
2020-04-17
Publication Date
2024-11-05
Estimated Expiration
2040-04-17

AI Technical Summary

Technical Problem

Current 4D scanning transmission electron microscopy (STEM) methods struggle with elemental identification in thick, heterogeneous biological samples due to asynchronous readout modes, slow frame rates, and inefficiencies in electron interaction detection, leading to distorted and noisy images.

Method used

Implementing a monolithic active pixel array sensor (MAPS) with a global shutter readout mode and pixel array detectors to synchronize electron beam scanning with image acquisition, allowing for high-speed, synchronized detection of elemental composition in thick biological samples.

Benefits of technology

Enables high-resolution, synchronized imaging of elemental composition in thick biological samples, reducing distortion and noise, and improving the efficiency and throughput of elemental identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577341000004
    Figure 0007577341000004
  • Figure 0007577341000005
    Figure 0007577341000005
  • Figure 0007577341000006
    Figure 0007577341000006
Patent Text Reader

Abstract

This disclosure relates to a transmission electron microscopy method for evaluating biological materials. According to embodiments, this disclosure further relates to an apparatus for identifying the structure and / or elemental composition of a sample using a 4D STEM, the apparatus comprising a direct impact detector operating with global shutter readout, and a processing circuit configured to acquire images of multiple bright-field disks using either a continuous or discontinuous array of detector pixel elements, correct for distortion in the images, align each image in the multiple images based on the centroid of the bright-field disks, calculate the radial profile of the images, normalize the radial profile by magnification, calculate the rotationally averaged edge shape of the bright-field disks, and determine the elemental composition in the sample based on the characteristic shape of the bright-field disk edge corresponding to each location in the sample.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 836,555, filed April 19, 2019, the teachings of which are incorporated by reference in their entirety for all purposes.

[0002] Government Support Statement This invention was made with Government support under Award No. DE-SC0018493 awarded by the Office of Science, Department of Energy. The Government has certain rights in this invention.

[0003] The present disclosure relates to scanning transmission electron microscopy and characterization of biological materials. Further, the present disclosure relates to a monolithic active pixel array sensor operating in a synchronous readout mode. The present disclosure also relates to the operation of the sensor in a compressed sensing readout mode. [Background technology]

[0004] Over the past decades, optical fluorescence microscopy has been the primary method for biologically observing cells with a resolution of hundreds of nanometers. This technique uses autofluorescent proteins, or small artificial fluorescent probes, to selectively label cellular components so that their location can be visually identified within a sample. Such specific structural information is important for the development of biochemistry. However, the wavelength of visible light limits the resolution of optical fluorescence microscopy to 200 nanometers. Although super-resolution techniques have pushed the resolution of fluorescent dye labels to higher resolutions, it is not possible to survey an entire sample with nanometer-scale resolution using optical microscopy. Therefore, many important cellular features, including membrane separations and the individual locations of macromolecular assemblies, cannot be clearly seen.

[0005] Thus, higher resolution (e.g., nanoscale) observation of biological microstructures can be facilitated by transmission electron microscopy (TEM). However, unlike the lower resolution technique optical fluorescence microscopy, the ability to label and identify different types of cellular components in a single image is minimal in conventional TEM. Thus, TEM has represented a trade-off between 100-fold improvement in resolution and the inability to label and identify functional cellular components.

[0006] For this purpose, the combination of optical fluorescence microscopy and TEM, called correlative light-electron microscopy (CLEM) when applied to the same specimen, can address this deficiency, but remains tedious and limited by the fact that fluorescent labels can still only be visualized at low resolution.

[0007] Recent developments include "multicolor EM," which selectively uses tagging with lanthanide ions and filtering by electron energy loss (EFTEM) to generate data similar to multicolor fluorescence microscopy, but at up to 100x magnification. For each protein or cell region of interest, diaminobenzidine (DAB) conjugated with a specific lanthanide metal (La+3, Ce+3, or Pr+3) can be locally deposited, and the exact location of the lanthanide metal is identified by collecting multiple EFTEM images at the characteristic core-loss edge of each element to obtain an elemental map. Although this technique is intended to reveal new structural information, it is often inefficient because it relies on a very small fraction of electrons interacting with the sample and accidentally losing a specific amount of energy. Furthermore, each tag must be acquired in a separate image, and a conventional TEM bright-field image must also be acquired. This inefficiency makes the technique tedious, time-consuming to perform, and produces noisy images that are difficult to interpret unambiguously, especially for thick cell samples.

[0008] Scanning transmission electron microscopy (STEM) is an alternative to traditional TEM bright-field and EFTEM imaging. Rather than illuminating an entire area of ​​interest on a sample and collecting image data, STEM uses a focused electron beam in a raster across the area of ​​interest on the sample. Various detectors are used to measure the interaction of the electron beam with each raster point on the sample, resulting in one or more images of the sample. These detectors are generally monolithic, so that each detector measures a single value, e.g., the number of electrons scattered by the material into a particular range of angles.

[0009] Recently, four-dimensional STEM (4D STEM) has been developed as an extension of conventional STEM methods. In 4D STEM, monolithic detectors are replaced or complemented by pixel array detectors consisting of thousands to millions of independent detector elements, allowing more complex analyses, including elucidation of local electromagnetic fields and crystal distortions within a sample. However, elemental identification has not been possible with 4D STEM until now, due to several deficiencies of current techniques and methods, especially in relatively thick samples such as those used in biology.

[0010] First, conventional electronic active pixel array detectors only operate in rolling shutter readout mode (i.e., asynchronous mode), where pixels across the array are acquired and readout at slightly different times. In many detectors, the readout proceeds sequentially across the rows of the pixel array. Rolling shutter readout suffers from distortion issues with fast moving objects and short flashes of light. For example, rolling shutters can cause wobble, skew, spatial aliasing, and temporal aliasing. In 4D STEM, rolling shutters pose significant problems. With rolling shutter readout, different groups of pixels (rows) have different timestamps, so it is not possible to synchronize the readout of the entire pixel array (all rows) with a single external event, such as the movement of the STEM probe over the sample.

[0011] Second, conventional electronic active pixel array detectors are generally too slow to acquire 4D STEM data of large sample areas without the deleterious effects of sample drift. The frame rate of most pixel array detectors can be increased by reducing the number of pixels read out each frame, for example so that only the central 25% of the pixels are read out. Ignoring some portion of the pixels at the periphery of the pixel array increases the frame rate, but it also reduces the range of scattering angles collected in the experiment. For 4D STEM methods that utilize a wide range of scattering angles, it is not practical to select a smaller region of interest in the pixel array to increase speed.

[0012] Third, existing methods in 4D STEM are not sensitive to elemental composition in thick, heterogeneous samples, such as those frequently used in biological applications. Elemental identification in STEM is facilitated by "Z-contrast," a single value based on the number of electrons scattered by the sample at large scattering angles (called the high-angle dark field region). Although Z-contrast is effective for observing crystalline samples with uniform atomic columns, the elemental composition of a mixture cannot be identified using this simple metric.

[0013] The present disclosure describes methods that address the deficiencies of current methods noted above.

[0014] The foregoing "Background" discussion is intended to generally provide a background to the present disclosure. The inventors' work within the scope of the Background, as well as aspects of the discussion that may not otherwise be considered prior art at the time of filing, are not admitted, expressly or impliedly, to be prior art to the present invention. Summary of the Invention [Problem to be solved by the invention]

[0015] The present disclosure relates to systems, devices, and methods for identifying elemental composition using 4D STEM.

[0016] According to an embodiment, a method for determining material properties of a sample using a 4D STEM includes directing an electron beam from an electron source towards the sample, detecting electrons from the electron beam that interact with the sample using an electron detector including an array of detector elements, acquiring images of a plurality of bright field disks and surrounding dark field diffraction based on the detected electrons from the electron detector, and determining an elemental composition of the sample based on the acquired images of the plurality of bright field disks and surrounding dark field diffraction.

[0017] According to an embodiment, a method for compressed readout using an electron microscope includes generating a row address table including a first set of row addresses and frame heights, the row address table including a truncated number of row addresses that is less than a maximum number of row addresses possible within a frame height, transmitting each row address of the truncated number of row addresses in the row address table to a controller, the controller configured to control an electron source to direct an electron beam toward a sample to read out charge from an electron detector including an array of detector elements, the electron detector configured to detect electrons in the electron beam, controlling the controller to read out according to the transmitted row addresses, determining whether the last row address in the row address table has been read out, and if it is determined that the last row address in the row address table has not been read out, controlling the controller to increment a pointer to the next row address in the row address table and read out according to the next row address, and repeating the control and determination until the last row address in the row address table has been scanned.

[0018] According to an embodiment, a system for determining material properties of a sample includes an electron detector including an array of detector elements configured to convert charge from impinging electrons into an electronic signal, and processing circuitry configured to read out the electronic signal from each detector element of the detector array in a global shutter mode.

[0019] The foregoing paragraphs have been provided by way of a general introduction and are not intended to limit the scope of the following claims. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0020] A more complete understanding of the present disclosure and many of the attendant advantages will be readily obtained as the disclosure becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings.

[0021] [Figure 1] FIG. 1 is a schematic diagram illustrating a system for image-wise detection of charged particles, according to an example.

[0022] [Diagram 2] FIG. 1 is a schematic diagram illustrating a cross section of a monolithic active pixel array sensor (MAPS) according to an example.

[0023] [Diagram 3] FIG. 1 is a schematic diagram showing a cross-section of a single pixel of a four-transistor (4T) sensor for front-side illumination (FSI), according to an example.

[0024] [Figure 4] FIG. 2 is a schematic diagram showing a cross section of a single pixel for back side illumination (BSI), according to an example.

[0025] [Figure 5A] 1 is a schematic diagram illustrating an example image captured by the system, according to one example.

[0026] [Figure 5B] FIG. 5B is a schematic diagram showing the pixel intensity profile of a 36-pixel wide vertical stripe in the center of each bright-field disk of the image in FIG. 5A.

[0027] [Figure 6] 1 is a flow chart of a method for observing a sample, according to an example.

[0028] [Figure 7A] According to an example, an image of a TEM beam stop acquired with 25% subsampling is shown.

[0029] [Figure 7B] 10B shows an image of the TEM beam stop of FIG. 10A including the unknown row and the full field of view of 4096×4096, based on an example.

[0030] [Figure 8A] 1 is a flow chart of a method for a compressed read mode of one frame according to an example.

[0031] [Figure 8B] 4 is a flow chart of a method for compressed readout mode of multiple frames having identical rows according to an example.

[0032] [Figure 8C] 13 is a flow chart of a method for compressed readout mode of multiple frames with unique rows according to an example.

[0033] [Figure 9A] Schematic of sample preparation for labeling mitochondria (left) and nuclear membrane (right) according to an example.

[0034] [Figure 9B] FIG. 1 is a schematic diagram of multicolor electron microscopy (EM) using the EFTEM technique, based on an example.

[0035] [Figure 9C] FIG. 1 is a schematic diagram of single-shot multi-color EM using 4D scanning transmission electron microscopy (4D STEM) based on an example.

[0036] [Figure 10] 1 shows images taken during a 4D STEM acquisition according to an example.

[0037] [Figure 11] 1 is a graphical representation of a radial profile of an image taken during a 4D STEM acquisition, according to an example.

[0038] [Figure 12] 1 is a flow diagram of a method for processing images acquired using 4D STEM, according to an example.

[0039] [Figure 13A] 1 is an image of a single frame acquired using a 4D STEM showing a central bright field disk and surrounding dark field region at a single probe position on a sample, according to an example.

[0040] [Figure 13B] 1 is an image of a sparsity map showing areas of a frame where primary electrons are not sparse enough to perform electron counting, according to an example.

[0041] [Figure 13C] 1 is an image of a frame after electron counting in a sparse region of the frame and scaling by the average pixel value for one primary electron at the detector, according to an example.

[0042] [Figure 13D] 13 is an image of a brightfield disk reflecting a nearby probe position and fitted to an ellipse to determine the position and elliptical distortion of the brightfield disk, according to an example.

[0043] [Figure 14A] 13 is a reconstructed image of a bright field using the total integrated intensity of the bright field disk at each probe position on the sample, according to an example.

[0044] [Figure 14B] 13 is a dark-field reconstructed image using the total integrated intensity outside the bright-field disk up to the edge of the pixelated detector, according to an example.

[0045] [Figure 14C] 1 is a reconstructed image where each pixel represents the amplitude of the centroid of a bright field disk, according to an example.

[0046] [Figure 14D] 1 is a reconstructed image where each pixel represents the angle of the center of gravity of a bright field disk, according to an example.

[0047] [Figure 15] 1 is an example image of an elemental map of cerium using EFTEM.

[0048] [Figure 16] 1 is a bright field reconstructed image with cerium locations and gold locations according to an example.

[0049] [Figure 17] 1 is a representation of computer hardware controlling components of a system according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0050] The terms "a" or "an," as used herein, are defined as one or more than one. The term "plurality," as used herein, is defined as two or more than two. The term "another," as used herein, is defined as at least a second or more. The terms "including" and / or "having," as used herein, are defined as "comprising" (i.e., open language). Reference anywhere in this document to "one embodiment," "certain embodiment," "an embodiment," "an implementation," "an example," or similar terms means that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the disclosure. Thus, the appearances of such expressions in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, particular features, structures, and characteristics may be combined in any suitable manner in one or more embodiments without limitation.

[0051] Cell biologists can introduce labels into living cells that migrate to specific regions, or attach the living cells themselves to specific reactants, products, catalysts, etc. Using imaging techniques that allow selective viewing of the labels, scientists can glean information about structure / function relationships in the cell body. Transmission electron microscopy (TEM) is the primary method for observing the fine structure of biological cells. However, unlike the lower resolution technique of fluorescence microscopy, it is not possible to label and distinguish different types of cellular components in a single image using conventional TEM.

[0052] To improve the throughput, efficiency, and resolution of multi-color EM, a scanning transmission electron microscopy (STEM)-based multi-color EM method is described herein that uses a high-speed pixelated detector to capture signals from a significant number of primary electrons interacting with the sample.

[0053] This disclosure describes a system that includes a demonstration camera and a synchronous STEM scanner mounted on a STEM. In evaluating this system, a 4D STEM data set of a cell mitochondrial matrix sample can be collected, where the mitochondria can be labeled with cerium and contain gold nanoparticles. After correcting for distortions in the diffraction pattern, metrics can be developed to identify the cerium label and the gold nanoparticles, as described in detail below, while simultaneously generating bright-field and dark-field images of the sample at a resolution significantly higher than that possible using fluorescence microscopy, and at a throughput and image quality significantly better than that possible with previous EFTEM methods.

[0054] Achieving the necessary sensitivity with this approach involves pixelated detectors delivering synchronous (global shutter) readout of a very large number of pixels (at least 512 x 512 pixels) with a single electron sensitivity. Because biological cell observation requires a large sample field, the pixelated detector also needs to be fast enough to scan over a large portion of the sample before instabilities in the sample or microscope introduce deleterious artifacts into the resulting data.

[0055] To broaden the applicability of this technique in biological research, much faster detectors are required so that large sample areas can be observed in a reasonable time with negligible sample drift.

[0056] In general, the present disclosure relates to a method for identifying elemental composition in an electron microscope. In embodiments, the present disclosure relates to a method for identifying elements in a sample based on analysis of details of a convergent beam electron diffraction (CBED) pattern using a pixelated charged particle detector and recording the details of the CBED pattern.

[0057] In embodiments, the methods described herein may be performed directly by a 4D STEM-based controller or by the processing circuitry of a 4D STEM.

[0058] To achieve the above, modifications to the detector include modifying the camera firmware so that the detector can either receive or generate timing pulses that are synchronized with the frames. When the camera operates as a master, it generates a pulse that is received by the scanner to move the scanner to the next probe position. The camera has adaptability so that it can control things like the pulse width and the exact timing of the pulse relative to the start time of the frame readout. The camera can also operate in a slave mode where it receives a pulse from the scanner that is used to start the frame readout. Adaptability exists to adjust the exact time that the readout occurs relative to the receipt of the pulse. Modifications to the detector further include enabling a global shutter readout. This capability, which is provided by design in the sensor and firmware, was developed and added to the camera to support this mode. The camera architecture is such that any number of new timing sequences can be developed and added to the camera as a firmware update. The camera supports switching between operational modes using a command interface.

[0059] Referring now to the drawings, in which like reference numerals refer to the same or corresponding parts throughout the several views, the following description relates to a monolithic active pixel array sensor (MAPS), a system including the MAPS, and a method of using the MAPS. The MAPS operates in a synchronous readout mode. The integration and reset times of each pixel of the MAPS across a two-dimensional (2D) pixel array are simultaneous. As described herein, a "pixel array" may also be understood to mean an array of sensing elements.

[0060] The MAPS described herein operates in a global shutter mode, in contrast to conventional active pixel array direct bombardment charged particle detectors, which operate only in a rolling shutter (asynchronous) readout mode, meaning that rows are reset and integrated sequentially one after the other (asynchronously). Global shutter detectors provide a means to capture an entire 2D image of a charged particle emitted from a single probe location without blurring artifacts.

[0061] Simultaneous integration of the entire image is also useful in charged particle observations whenever images of fast moving situations are recorded, such as in-situ electron microscopy (EM).

[0062] FIG. 1 is a schematic diagram showing a system 100 for imagewise detection of charged particles according to an example. The system 100 uses a global shutter (GS) readout mode. The system 100 includes a monolithic active pixel array sensor (MAPS) 102, a scan controller 104, a processing circuit 106, and an electron source 108. The system 100 may also include a sample holder (not shown). Again, a pixel array may be understood to be an array of detector elements. For simplicity, it is referred to as a "pixel array" in this specification. The MAPS 102 operates synchronously with the scan controller 104 by sending or receiving a synchronization signal that controls the position of a charged particle beam probe at the same frequency as the frame rate of the MAPS 102. Thus, the scan controller 104 is synchronized to the frame rate of the MAPS 102. A single sharp diffraction image is recorded at each probe position, for example, at the position of a scanning transmission electron microscopy (STEM) probe. Global shutters are useful in STEM and scanning electron microscopy (SEM) to prevent detector frames from mixing across multiple beam positions.

[0063] In one implementation, the frame rates can be 255fps, 484fps, 875fps, 1470fps, or 2227fps, with corresponding output sizes of 2048x2048, 1024x1024, 512x512, 256x256, and 128x128, respectively.

[0064] In one implementation, the sensor is configured to process 256x256 pixel global shutter readouts at approximately 20,000 fps, a speed that allows a 4096x4096 sample area to be covered in under 14 minutes.

[0065] The scan controller 104 is an adaptive and programmable scan controller that can operate or be operated by the MAPS 102 .

[0066] The global shutter pixel design is used in the MAPS direct impact charged particle detector, which is described further below.

[0067] MAPS simultaneously resets and acquires data. In one implementation, a hardware trigger signal corresponding to the start of acquisition of each frame is sent to the STEM scanning device, allowing precisely synchronized high-speed four-dimensional STEM (4D STEM) data acquisition.

[0068] In one implementation, system 100 can be a transmission electron microscopy (TEM) system (e.g., 4D STEM or ptychography) system. A combination of a global shutter (GS) MAPS and a probe position controller provides high speed scanning probe charged particle microscopy. System 100 can include a MAPS in the energy range from 200 keV to 1 MeV, as described later in this specification.

[0069] In one implementation, the system 100 can include a MAPS operating at energies less than 30 keV. For example, the system can be a SEM system (e.g., electron backscatter diffraction (EBSD), transmission Kikuchi diffraction (TKD), ptychography).

[0070] In one implementation, the system can be for in-situ microscopy to record video without rolling shutter artifacts. System 100 can be a TEM system operating at or above 120 keV using MAPS.

[0071] The radiation sensitivity of the MAPS detector is paramount, so a simple pixel design has been deemed necessary to maximize sensor lifetime. Even with such a simple pixel design, MAPS detectors have a finite lifetime and will require replacement after a period of use. In one implementation, the MAPS102 has a four-transistor (4T) pixel design that allows for a global shutter without sacrificing sensor lifetime. In other implementations, the MAPS102 may have pixels with more than four transistors.

[0072] Because electrons penetrate the large volume of the MAPS device, other MAPS global shutter implementations can suffer from a "leaky shutter" problem, meaning that charge for the next integration time can move from the charge collection / integration area (sometimes called pinned photodiode or PPD) to the charge storage area (sometimes called "sense node", "floating diffusion", or FD), resulting in additional noise and / or temporal blurring between multiple frames of the detector. The area of ​​the charge storage area (i.e., floating diffusion (FD)) relative to the charge collection / integration area (i.e., pinned photodiode (PPD)) is minimized. Reducing the volume ratio of the FD area to the PPD area is beneficial in minimizing the leaky shutter problem. In one implementation, the MAPS102 has a small FD resulting in a reduced volume ratio. A small FD leads to a low capacitance, which in turn leads to a high charge-to-voltage conversion rate.

[0073] As electrons pass through the metal layer, the small FD capacitance is overcome by incorporating a second charge storage capacitor to achieve HDR mode without sacrificing aperture ratio. In one implementation, the second charge storage capacitor is a metal-insulator-metal (MIM) capacitor. The MIM capacitor improves shutter efficiency in global shutter mode by acting as a metallic light shield for FD when photons are included, but blocks electrons.

[0074] MAPS102 has a large charge depletion region for the PPD and a small charge depletion region for the FD. Therefore, the effective volume of the FD is reduced and the volume of the PPD is maximized. The large charge depletion region for the PPD and the small charge depletion region for the FD are achieved by controlling the levels of dopants in the regions associated with the FD and the PPD, respectively, as will be understood by those skilled in the art. For example, a highly doped p-well is used to reduce the volume of the FD. Although those skilled in the art may not do this due to the potential sacrifice of too much radiation hardness, as determined by the present inventors, the large charge depletion region for the PPD and the small charge depletion region for the FD have a minimal effect on radiation hardness, and experimental testing has shown that radiation hardness is not affected by this design.

[0075] Electrons in the 100 keV and higher energy range enter through the front passage and pass completely through the epitaxial silicon region, depositing charge along their entire trajectory. Electrons in the 100 keV and higher energy range, as determined by the inventors, behave more like red photons than blue photons in terms of where charge is generated. Electrons with energies below 30 keV and entering through the backside behave more like blue photons. As the electron energy decreases, the energy is deposited closer to the backside of the MAPS. In either of these cases, minimizing the volume of the FD improves shutter efficiency by reducing the likelihood that charge deposited by electrons entering the sensor will migrate to the FD instead of the PPD.

[0076] 2 is a schematic diagram showing a cross section of the MAPS 102 according to one example. The PPD 202 has a large area and volume and is shown to collect electrons from a large region including a region very close to the FD 206, shown in FIG. 2 by 204. The FD 206 has a small depletion region and collects only electrons created in a volume very close to the FD 206 (shown in FIG. 2 by 208). The contacts 210 to the MM 212 are minimized to minimize their contribution to the leaky shutter problem.

[0077] The front side of MAPS 102 is shown in FIG. 2 at 214. The back side of MAPS 102 is shown in FIG. 2 at 216. Region 218 contains many layers, not shown, involved in creating the transistors and interconnects that make up the pixel electronics. Electrons absorbed in region 218 do not create conduction band electrons (photocharges) that contribute to the signal. Epitaxial silicon region 220 (also called epi) generates and collects charge. Below epitaxial silicon region 220 is the substrate (not shown).

[0078] Microscopic electrons enter the device from either the front surface 214 or the back surface 216, as described later in this specification. The electrons interact with silicon in the epitaxial silicon region 220 and generate conduction band electrons. Unlike photons, microscopic electrons have much more energy and generate many conduction band electrons for each incident electron. The number of conduction band electrons generated depends on many factors, including the incident energy of the system (e.g., the microscopic electrons).

[0079] In one implementation, the accumulation area is limited to, for example, less than 5%, less than 2.5%, or preferably less than 1.25% of the charge integration area. The volume ratio is, for example, greater than about 1 / 200, 1 / 400, or preferably 1 / 800. If the shutter efficiency were determined entirely by the volume ratio, the sensor would have a shutter efficiency of 99.875% for a volume ratio of 1 / 800.

[0080] In one implementation, the charge accumulation area is limited to less than 0.8% of the charge integration area. The volume ratio is approximately 1 / 1200. If the shutter efficiency was determined entirely by the volume ratio, the sensor would have a shutter efficiency of 99.92%.

[0081] For electrons, the shutter efficiency scales with the volume ratio. The ratio is maximized by using larger pixels. The shutter efficiency can be estimated from the volume ratio and additional information about where photocharges are likely to be generated within the epi region. The estimates can be compared to measurements to check the validity of the assumptions.

[0082] 3 is a schematic diagram showing a cross section of a single pixel 300 of a 4T type sensor for front-side illumination (FSI) according to an example. Electrons with energy greater than 30 keV, preferably greater than 100 keV, enter the front of the pixel 300, pass through the circuit layer 302, and deposit charge throughout the epi layer 304 and substrate 306. Charges generated in the substrate 306 recombine due to the high doping in the substrate and do not contribute to the measured signal. Charges generated in the epi 304 are mostly collected by the PPD 308.

[0083] For high energy electrons (typically greater than 100 keV to >1 MeV), good shutter efficiency is achieved by minimizing the volume ratio of FD to PPD as described herein above and taking advantage of the fact that the microscopic electrons generate photocharge throughout the entire thickness of the epilayer 304 as determined by the inventors. The epilayer 304 has a thickness ranging from 5 microns to 18 microns. Additionally, the substrate is thinned leaving a layer that can range from a few microns to tens of microns from the backside. This prevents the electrons from backscattering within the substrate and re-entering the epilayer and contributing to noise by depositing additional charge in areas away from where the electrons originally entered.

[0084] 4 is a schematic diagram showing a cross section of a single pixel 400 for backside illumination (BSI) according to an example. Better sensitivity is achieved by introducing a signal from the backside for electrons with energies less than 30 keV. The fabrication method of pixel 400 includes bonding a conventional wafer to a handle substrate and removing the substrate from the sensor to expose the epi surface.

[0085] The absorption length of electrons in silicon below 30 keV range is on the order of 5 microns to less than 1 micron for 1 keV electrons, as determined by the inventors. The pixel 400 has a 5-6 micron epilayer and a shallow FD. High shutter efficiency with BSI is achieved because the generated charge carriers diffuse to the PPD rather than the FD.

[0086] In one implementation, the BSI MAPS is included in a TEM system operating in the 60keV to 120keV range with BSI detection.

[0087] To demonstrate the capabilities of MAPS102 as described herein, exemplary results are presented.

[0088] To assess the synchronization between the sensor frame rate and the movement of the STEM probe controlled by the scan controller 104, the scan controller 104 instructed the STEM probe to move back and forth between the empty sample area and the copper grid bars. The trigger delay of the sensor (i.e. the camera) was adjusted until the intensity difference between the bright field disk in the bright frame (i.e. when the STEM probe is over the empty sample area) and the bright field disk in the dark frame (i.e. when the STEM probe is over the copper grid bars) was maximized. This process is repeated in the rolling shutter readout mode and the global shutter readout mode.

[0089] FIG. 5A is a schematic diagram showing an example image taken by the system described herein. The TEM beam position was switched back and forth between the empty sample area and the copper grid bar in synchronization with the sensor frame rate. Image 502 is an image of the bright field disk when the beam was over the empty sample area using the rolling shutter technique. Image 504 is an image of the bright field disk when the beam was over the empty sample area using the global shutter technique described herein. Image 506 is an image of the bright field disk when the beam was over the copper grid bar using the rolling shutter technique. Image 508 is an image of the bright field disk when the beam was over the copper grid bar using the global shutter technique described herein.

[0090] Figure 5B is a schematic diagram showing the pixel intensity profile of a 36 pixel wide vertical stripe in the center of each bright field disk in the image shown in Figure 5A. A line of best fit to the pixels is shown to intersect with the actual data points. As can be seen in the image, the rolling shutter mode presents a reduced intensity difference between the bright and dark frames and an intensity gradient visible in the bright field disk. These artifacts are visible in the rolling shutter readout mode regardless of the camera trigger delay setting, indicating that the rolling shutter readout always records a mix of the two STEM probe positions.

[0091] Line 510 corresponds to the intensity profile of image 504. Line 512 corresponds to the intensity profile of image 502. Line 514 corresponds to the intensity profile of image 506. Line 516 corresponds to the intensity profile of image 508.

[0092] As shown by line 516 in Figure 5B, in the global shutter readout mode, there is very little intensity between the bright and dark frames, indicating that the frame readout from the detector is well synchronized with the STEM probe motion: each frame corresponds to exactly one STEM probe position.

[0093] 6 is a flow chart of a method for observing a sample, according to an example. In step 602, an electron beam is directed at the sample.

[0094] At step 604, a sensor is provided. The sensor can be any pixelated detector, but is preferably a direct electron detector, and more preferably a global shutter direct electron detector. The global shutter direct electron detector can be a back-illuminated MAPS 102 or a front-illuminated MAPS 102, as described herein above. The sensor is placed below the sample. The detector includes a number of pixels configured to detect electrons passing through the sample. In one embodiment, the sensor includes 4096×4096 pixels. In another embodiment, the sensor includes 1024×1024.

[0095] In step 606, data is read out from the detector in global shutter mode so that the MAPS frames are synchronized with the moving probe, for example a STEM or SEM probe.

[0096] At step 608, the output from the sensor is sent to a recording and / or display device. For example, a user interface can display images to a user and can receive commands from the user.

[0097] According to an embodiment, Figures 7A-7C relate to a method for customizing row / column readout of a pixel array and compressed sensing in MAPS 102. Advantages include enhanced sensitivity, speed, resolution, and field of view.

[0098] For complementary metal oxide semiconductor (CMOS) detectors used in EM (e.g., scintillator-coupled, direct detection, etc.), the camera frame rate can be increased by reducing the camera's field of view in a first dimension, e.g., the Y dimension. An image may contain multiple rows of pixels, and the rows may be indexed along the Y dimension, e.g., using a numeric address. In some scanning methods, a frame can be created by reading all or some of the rows in an image. A frame can be created by the camera reading out the rows sequentially. Because the row readout is sequential, reducing the number of rows read out reduces the time required to read out each frame, regardless of whether the MAPS102 is operating in a rolling shutter readout mode or a global shutter readout mode, thereby increasing the maximum frame rate of the camera. Because such detectors generally require consecutive row readouts, reducing the number of rows read out from the camera can also reduce the camera's field of view.

[0099] Described herein is a method to alleviate the above problems by reading randomly selected rows across the field of view and using inpainting algorithms or other methods to restore or account for missing information, providing a high frame rate and large field of view. The method can also rapidly change the rows or kernel rows selected in response to a determination of the scan range of the object, such that information from the image reconstruction can be used to change the selection of rows used in the acquisition of subsequent frames. This aspect of the method may alleviate the problem of potentially selecting suboptimal rows for subsampling based on unknown image content.

[0100] MAPS 102 may be arranged in multiple rows and columns. A number of adjacent rows may be grouped into a "kernel row" and a number of adjacent columns may be grouped into a "kernel column". Instead of reading all rows / columns from a sensor or subarray of contiguously arranged rows / columns of a detector, a number of randomly selected non-contiguous arbitrary rows / columns may be read out from the detector. Thus, some rows / columns may be ignored. The ignored rows / columns may be read out with a constant pixel value or omitted in the image output from the detector. In some cases, the missing image information may be restored after image capture using image restoration methods. This method may enable an increase in frame rate without reducing the field of view. This strategy may also use kernel rows or columns instead of individual rows or columns. This is useful for enabling compressed sensing applications to observe charged particles or photons.

[0101] The disclosed method allows imaging systems to provide higher frame rates than would naturally be possible by subsampling or sparsely sampling information and then repairing the missing information using an algorithm. The disclosed method can also be used to reduce the amount of data that needs to be transmitted while still retaining a sufficient amount of information for the intended purpose. This can be useful, for example, when transmission bandwidth is limited, data storage space is limited, or both. This approach is applicable to the EM domain when dynamic processing is considered (high frame rates are desired). Similarly, this approach can be advantageous in any imaging domain where high frame rates or reduced data size are of interest.

[0102] For some sampling methods, the subsampling is periodic and therefore not suitable for restoration (only interpolation). Furthermore, periodic subsampling may not allow for rapid changes in the subsampling time, i.e. new values ​​are written to control registers without interrupting acquisition and acquisition is restarted.

[0103] In an embodiment, the scan controller method of subsampling is complementary to the image sensor subsampling. It is possible to combine both methods, resulting in further data reduction and increased frame rate.

[0104] FIG. 7A shows an image of a TEM beam stop acquired with 25% subsampling on the detector, according to an example. In an embodiment, an image may be acquired that is a 4096×1024 image with compressed information read out from the camera, increasing the frame rate by a factor of four. To meet the need for higher frame rate observations, a new undersampled compressed readout mode, called Arbitrary Kernel Row Addressing (AKRA) or Arbitrary Row Addressing (ARA), can be utilized. For example, an image of a TEM beam stop can be acquired in AKRA readout mode. In AKRA readout mode, the user can specify any combination of kernel rows to read out from the detector while omitting all other kernel rows. For example, a kernel row may contain 8 unbinned pixels in the height direction. In ARA readout mode, individual rows can be specified. Reading out fewer rows can significantly increase the frame rate of the detector, but the field of view on the sample may not be reduced since the kernel rows are distributed throughout the MAPS102.

[0105] FIG. 7B shows an image of the TEM beam stop of FIG. 7A, including the missing row and the full field of view of 4096×4096, based on an example. Again, the image of FIG. 7A can be acquired in AKRA mode with 25% subsampling, and FIG. 7B shows an extension of the subsampled image. Depending on the application, the "missing" pixels in the AKRA (or ARA) image can be restored using an inpainting algorithm, or measurements can be made directly from the actual rows read out, as long as the coordinates of each image are properly taken into account. Since the selection of the kernel rows is pseudorandom, the edges of the beam stop in FIG. 7A are jagged rather than uniform staircase steps, and the light bands in FIG. 7B are randomly distributed. In other words, random sampling is optimal for effective inpainting.

[0106] In an embodiment, design features of MAPS102 that may enable this mode of operation include: (1) an address decoder for rows (or kernel rows) that allows selection of row addressing for readout; (2) the address decoder can receive any address (rows can be addressed in any order, any number of rows can be omitted or read out in groups, etc.); and (3) a high-speed interface that allows row addresses to be sent to MAPS102 in a pipelined manner so that the next row to be addressed can be specified while the current row is being read out.

[0107] In an embodiment, camera design features that may enable a continuously scanned full sampling readout mode of operation include: (1) a table to read out, containing the addresses of each row (or kernel row) in order; (2) a termination character at the end of the table; (3) a pointer that is advanced within the table; and (4) a register containing the total number of rows to be read out.

[0108] In an embodiment, the full sampling readout mode of one frame involves incrementing a pointer in a table of row addresses with the total number of rows to be read (frame height). For each row the pointer is incremented and the new row address is sent to the camera. This procedure continues until all rows are read. If an end of line character is reached before the total number of rows is exceeded, the pointer is reset to the first entry in the row address table and the process continues until the total number of rows is reached. This feature is particularly advantageous in facilitating the compressed readout described herein.

[0109] In an embodiment, camera design features that may enable a random row mode of operation (compressed mode) (similar to the one above) include: (1) a programmable bit (register) to indicate full sampling mode or compressed readout mode; (2) a table containing the address of each row (or kernel row) that the user specifies for readout; (3) a way to read values ​​into the table (a series of registers - table location and row address); (4) a way to read the values ​​contained in the table; (5) a termination character at the end of the table; (6) a pointer that is advanced within the table; and (7) a register containing the total number of rows to be read. For example, the row addresses could be any row, all rows, or a minimum of two rows specified in any order. The table length can be anything, limited only by the amount of available memory. For example, 64000 addresses could be available. This is a different table than the fast switching one used for sequential readout described above.

[0110] FIG. 8A is a flow chart of a method for a compressed readout mode of one frame according to an example. In step 1102, a register is set to perform the compressed readout mode. In step 1104, if the row address table has not been loaded, the method proceeds to step 1106. For example, the row address table may include an arbitrary row address and a frame height. In step 1106, the row address table is loaded. If the arbitrary row address table has been loaded, the method proceeds to step 1108. In step 1108, the address of the row to be scanned is transmitted to the camera or electron source. In step 1110, the camera scans the transmitted row and the pointer is incremented to the next row to be scanned in the row address table. In step 1112, if the frame height has not been completely scanned, the method returns to step 1110 and the next row is scanned. If the frame height has been completely scanned, the method proceeds to step 1114. In step 1114, the end of the frame is reached and the scan ends.

[0111] Table 1 shows an example of arbitrary row addresses for the compressed read mode method described in FIG. 11A.

[0112] [Table 1]

[0113] FIG. 8B is a flow chart of a method for compressed readout mode of multiple frames with identical rows, according to an example. In step 1202, a register is set to perform compressed readout mode. In step 1204, if the row address table is not loaded, the method proceeds to step 1206. For example, the row address table can contain an arbitrary row address and a frame number, and the same row address may be scanned for multiple frames until the frame number is reached. In step 1206, the row address table is loaded. If the arbitrary row address table is loaded, the method proceeds to step 1208. In step 1208, the address of the row to be scanned is transmitted to the camera or electron source. In step 1210, the camera scans the transmitted row, and the pointer is incremented to the next row to be scanned in the row address table. In step 1212, if the frame height has not been completely scanned, the method returns to step 1210 and the next row is scanned. If the frame height has been completely scanned, the method proceeds to step 1214. In step 1214, the end of the frame is reached and the method proceeds to step 1216. In step 1216, the frame count is decremented and the pointer is reset to the first address (i.e., the first row scanned). If the frame count has not been reached in step 1218, the method proceeds to step 1208 for another compression scan. If the frame count has been reached, the scan ends.

[0114] Table 2 shows an example of arbitrary row addresses for the compressed read mode method described in FIG. 8B.

[0115] [Table 2]

[0116] FIG. 8C is a flow chart of a method for a compressed read mode of multiple frames with unique rows, according to an example. In step 1302, a register is set to perform the compressed read mode. In step 1304, if the row address table has not been loaded, the method proceeds to step 1306. For example, the row address table can include any row address and a termination character, and the row addresses can be scanned for multiple frames until the number of frames is reached and the row addresses for each frame are unique. The termination character can be used to signal the end of the scanning method, and a new frame is scanned until the termination character is reached. In step 1306, the row address table is loaded. If the any row address table has been loaded, the method proceeds to step 1308. In step 1308, the address of the row to be scanned is transmitted to the camera or scanning device. In step 1310, the camera scans the transmitted row, and a pointer is incremented to the next row to be scanned in the row address table. In step 1312, if the frame height has not been completely scanned, the method returns to step 1310 and the next row is scanned. If the frame height is completely scanned, the method proceeds to step 1314. In step 1314, the end of the frame is reached and the method proceeds to step 1316. In step 1316, a pointer is set to the first address for the next frame (i.e., the first row to be scanned). The first address, and all addresses for the next frame, may be different from the addresses of the previously scanned frame. Compared to the method of FIG. 8B, this method does not use the same table for each scan. Rather, a large table can be generated for all frames to be scanned, with each frame having a unique address, or a new table can be generated after each frame is scanned. In step 1318, if the end character in the row address table has not been reached, the method proceeds to step 1308 and another compression scan is performed. If the end character is reached, the scan ends.

[0117] Table 3 shows an example of arbitrary row addresses for the compressed read mode method described in FIG. 8C.

[0118] [Table 3]

[0119] STEM has been used to view biological samples and has been shown to produce sufficient contrast at low electron energies to allow viewing without contrast agents (stains), and STEM has been used with cryogenically cooled samples.

[0120] Spectroscopic techniques have also been developed for STEM, including electron energy loss spectroscopy (EELS) and energy dispersive X-ray spectroscopy (EDX, EDXS, EDS, XEDS). These methods allow the identification of elements present in a sample. EELS, and the related non-STEM energy filtered TEM (EFTEM) imaging method (also called EFTEM), use a very small fraction of the incident electron beam and are therefore very inefficient. Similarly, EDX is very inefficient. A consequence of this inefficiency is that the sample must be exposed to high levels of electron irradiation to obtain useful results. A second consequence is that these measurements can be very time consuming, and the sample or microscope may not be stable enough to collect data from large areas.

[0121] It is an object of the present invention to provide an efficient means of collecting elemental information in a single STEM scan. It is a further object of the present invention to reduce the amount of beam that a sample is subjected to in order to produce an elemental map. It is a further object of the present invention to reduce the time required to produce elemental maps from a sample, particularly from large areas of biological or other samples that are susceptible to beam damage.

[0122] In conventional STEM, dedicated bright-field (BF), annular dark-field (ADF), and high-angle annular dark-field (HAADF) detectors are used for data acquisition, typically resulting in three data points per scan coordinate on the sample. In 4D STEM, pixelated detectors replace all of the conventional STEM detectors (potentially except perhaps the HAADF detector). The pixelated detectors capture a two-dimensional image for each scan position on the sample. In this disclosure, a large camera area (1024×1024 pixels) was used, resulting in over one million (1,048,576) pixels per scan coordinate on the sample. This is more than six orders of magnitude more data than conventional STEM.

[0123] 9A-9C, the present disclosure includes a fabrication method for multi-color EM methods.

[0124] FIG. 9A shows the preparation of a sample labeling mitochondria (left) and nuclear membrane (right), FIG. 9B shows the multicolor EM method with EFTEM, and FIG. 9C shows single-shot multicolor EM with 4D STEM based on an example. First, a standard calibration grid with graphitized carbon and gold islands can be used as an initial test sample for 4D STEM. Then, a sample of cell mitochondrial matrix labeled with Ce-DAB, ruthenium tetroxide, and 30 nm gold particles can be used for the development and evaluation of single-shot multicolor EM methods. The preparation of the biological sample can be performed as shown.

[0125] High-speed pixelated detectors can be used to capture a sequence of images during scanning transmission electron microscopy (STEM) acquisition, with a one-to-one correspondence between the image and the electron probe position on the sample.

[0126] The method will now be described with reference to FIG. 12 taking into account FIGS.

[0127] The acquired bright field disk image is generally characterized by an intense disk (the "bright field disk") surrounded by sparse electron events (the "dark field region"). A typical image in such a data set is shown in Figure 10. In step 1502, an image of the bright field disk can be acquired.

[0128] Small misalignments, or aberrations, within the electron microscope can cause the bright field disk to shift position from one image to the next, and can also distort the bright field disk so that it is not perfectly circular. These misalignments and distortions can introduce errors into the analysis and comparison of the bright field disks in the different images within each data set.

[0129] These shifts and distortions can therefore be corrected so that the bright field disks in all images in a dataset are consistent in overall shape and position within each image.

[0130] Referring again to FIG. 12 and step 1504, the brightfield disks can be binarized by applying a threshold to each image to correct for shifts and distortions. In step 1506, the brightfield disks from several images corresponding to multiple local regions of the sample can be combined (by adding or averaging) to reduce noise or distortions caused by the sample. Then, in step 1508, an ellipse can be fitted to the edges of the brightfield disks and geometric distortions are corrected using standard distortion correction methods. A linear transformation can then be applied to each of the original images such that the center of gravity of the binarized brightfield disks of each image (or images) in the dataset is at the same location for all images in the dataset.

[0131] In step 1510, a radial profile corrected for shear and distortion is calculated for each image in the dataset. In step 1512, the radial profile of each corrected image in the dataset can be normalized by the average intensity of the bright field disk (i.e., the value of the radial profile near the zero point) or the average intensity of the dark field region (i.e., the value of the radial profile at the maximum radius), or a combination of the two.

[0132] An exemplary radial profile, normalized by the average intensity of a bright field disk, is shown in FIG.

[0133] For each image, the goal is to determine features of the bright field disk edge shape for each image so that the edge shapes for each image in the same series can be compared and classified by similarity. In one embodiment, each image in the 4D STEM data stack is classified according to the average gradient of the bright field disk edge shape. In another embodiment, each image in the 4D STEM data stack is classified according to the maximum gradient of the bright field disk edge shape. In another embodiment, each image in the 4D STEM data stack is classified by comparing the bright field disk edge shape to a theoretical or empirically derived model of edge shape for a particular elemental composition. In another embodiment, each image in the 4D STEM data stack is classified by machine learning techniques using the bright field disk edge shape.

[0134] As described above, the gradient of the normalized radial profile at the edge of the bright field disk can be calculated for each image, in step 1514. The gradient in each image can then be used to characterize each corresponding location on the sample, in step 1516. This can be done either by creating a reconstructed image of the sample using the gradient values ​​at each point on the sample, or by sorting the gradient values ​​into a number of discrete groups to label each location on the sample.

[0135] According to an embodiment, the gradient of the bright field disk may also be combined with other information of the images in the data set, such as the total intensity or average intensity of some or all of the dark field regions.

[0136] According to an embodiment, to improve the signal-to-noise ratio (SNR) of the dark field regions in each image, an electron counting algorithm can be applied to the sparse dark field regions such that each "blob" in the dark field region is normalized in intensity and / or size.

[0137] 12, the MAPS 102 is used to acquire images of a Convergent Beam Electron Diffraction (CBED) pattern, also known as a Bright Field (BF) disk in Scanning Transmission Electron Microscopy (4D STEM). In addition, the detector acquires information from the dark field around the BF disk.

[0138] To correct for the elliptical distortions that frequently occur in the CBED pattern / BF disk, an ellipse or other shape can be fitted to the edge of the CBED pattern / BF disk, either in individual frames or combinations of many frames, across the entire data set or for each probe position, and this fit is used to warp the image to produce a circular disk so that a rotational average can be calculated, or it is used to directly calculate the average following the ellipse or other shape.

[0139] Before averaging or calculating the radial average, a registration step can be used to register the centers to the same coordinates to compensate for possible variations in the CBED figure / BF disk center location for each probe position. This registration step can consist of finding the center of gravity of the BF disk.

[0140] A rotational profile of the CBED figure / BF disk, extending from the optical axis past the edge of the BF disk into the dark field region, can be calculated for each frame or combination of multiple frames. The rotational profile can be normalized by a factor such as the average intensity of the CBED figure / BF disk.

[0141] The gradient of the rotationally averaged CBED pattern / BF disk edge shape can be calculated and used by itself or in conjunction with other information, such as dark field intensity, bright field intensity, or other values ​​derived from BF and DF, to determine the properties of the sample at each probe position.

[0142] This method is useful for identifying the elemental composition of a sample using 4D STEM (single shot multicolor EM).Other methods for identifying the elemental composition of a sample based on the shape of its radial profile may include (1) classifying the radial profile of the edge of the brightfield disk by similarities using mean square difference, dynamic time warping, or Fréchet distance, (2) classifying the radial profile of the edge of the brightfield disk by K-means or other machine learning classification methods, and (3) comparing the radial profile of the edge of the brightfield disk to a dictionary of features of the radial profiles of various elements.

[0143] 13A through 16D are illustrative diagrams of the analysis of 4D STEM data from MAPS102, according to one example. FIG. 13A is a single from the detector showing the central brightfield disk and surrounding darkfield regions at one STEM probe position on the sample. The pixel count can be used as a threshold for primary electron events in the neighborhood of each frame to calculate a "sparseness map" shown in FIG. 13B, which shows the regions of the frame where the primary electrons are not sparse enough to perform electron counting. After performing electron counting in the sparse regions of each frame, the non-sparse regions of each frame can be scaled by the average pixel value for one primary electron at the detector. The resulting frame is a composite of electron counting and integration mode data, as shown in FIG. 13C, optimizing the signal-to-noise ratio. Brightfield disks at nearby STEM probe positions are averaged and fitted to an ellipse to determine the position and elliptical distortion of the brightfield disk, as shown in FIG. 13D. A circular integration can then be performed based on this.

[0144] Figures 14A to 14D are illustrative diagrams of 4D STEM reconstructions of graphitized carbon, based on an example. Figure 14A is a bright-field reconstruction using the total integrated intensity of the bright-field disk at each probe position on the sample. Figure 14B is a dark-field reconstruction using the total integrated intensity outside the bright-field disk up to the edge of the pixelated detector. Figure 14C is a reconstruction where each pixel represents the amplitude of the centroid of the bright-field disk. At sample positions where the primary electrons have a preferred scattering direction, the amplitude of the centroid is large. Similarly, Figure 14D is a reconstruction where each pixel represents the angle of the centroid of the bright-field disk. Red corresponds to 0° (to the right).

[0145] 15 and 16 provide non-limiting illustrative results using the 4D STEM system of the present disclosure, based on one example.

[0146] In an exemplary embodiment, samples of cell mitochondrial matrix labeled with cerium-DAB, ruthenium tetroxide, and 30 nm gold particles can be used. EFTEM can be used as a reference for validation of new 4D STEM-based techniques. EFTEM was performed using a JEM-3200EF TEM (JEOL, Japan) operated at 300 kV and equipped with an in-column Ω-type filter and a LaB6 electron source.

[0147] EFTEM images of the front and rear ends can be obtained using a 30 eV wide slit. Conventional bright field TEM images and EFTEM elemental maps can be obtained using a conventional CCD camera. As a result, elemental maps could be generated.

[0148] STEM imaging was performed using a Titan Halo (Thermo Fisher, Waltham, MA) operated at 300 kV and equipped with brightfield and HAADF STEM detectors, and a high-current XFEG electron source. 4D STEM data were acquired using a spot size of 8 with a camera length of 150 mm on a DE-16 direct detection camera (Direct Electron) with hardware frame synchronization to a DE-FreeScan STEM scanner (Direct Electron). The DE-FreeScan can acquire STEM data using any or subsampled scan geometry, but was operated in conventional full raster scanning mode. The DE-16 was operated at 342 frames per second (fps), no hardware binning was performed, and the readout area was 1024 × 1024 pixels. 4D STEM data were processed using the newly developed, GPU-accelerated DE-4DExplorer software (Direct Electron). Briefly, the software reads the 4D STEM data stack from disk and performs standard flat-field (darkness and gain) corrections. The software then calculates a sparsity map for each frame, which is a binary mask that corresponds to the regions of the frame where the number of primary electrons per pixel is low enough to be processed using electron counting. Based on the sparsity map, the software performs an electron scaling for sparse regions and scales the intensity of non-sparse regions based on the average pixel intensity per primary electron. Thus, pixel intensities in the final processed frame approximately correspond to the actual number of primary electrons incident on each pixel in each frame.

[0149] Using this data, 4D STEM analysis was completed by calculating the integrated intensity (total number of primary electrons) in all possible annular disks from the diffraction center to the edge of the detector. However, the bright-field disks were not exactly located at the center of our pixelated detector (i.e., the center moved slightly at different STEM probe positions and the BFD had slight ellipticity). To correct for these diffraction distortions, the bright-field disks were averaged over a 16 × 16 STEM pixel area on the sample and an ellipse was fitted to the edge of the average bright-field disk. The bright-field disks were then translationally aligned for each frame and their ellipticity was corrected before further processing to generate a reconstruction of the sample.

[0150] After correcting for distortions in the diffractograms, the cerium labels and gold nanoparticles were identified based on their radial scattering profiles. For example, the radial scattering profiles can be normalized by the average bright-field signal so that the scattering profiles of all sample pixels start out consistent. The annular bright-field and dark-field portions of each scattering profile curve can then be fitted to a linear equation. Using the parameters of such fitted linear equations, a K-means algorithm can be performed to classify the sample pixels into four different groups. The results were similar to the EFTEM results shown in Figure 15, but the 4D STEM results showed higher contrast as shown in Figure 16. Also, using the same data, it was possible to simultaneously generate bright-field and dark-field images of the sample with significantly higher resolution than was possible using fluorescence microscopy. Because all of these reconstructions were generated from a single STEM acquisition, the 4D STEM-based multicolor EM method has a significantly better throughput than was possible with previous EFTEM-based multicolor EM methods.

[0151] The hardware of the controller according to an exemplary embodiment will now be described with reference to FIG. 17. In FIG. 17, the controller includes a CPU 2000 for executing the processes described above. Process data and instructions can be stored in memory 2002. These processes and instructions can also be stored in a storage medium disk 2004, such as a hard drive (HDD) or a portable storage medium, or can be stored remotely. Furthermore, the claimed advances are not limited to the form of computer readable media on which the instructions of the inventive process are stored. For example, the instructions can be stored on a CD, DVD, flash memory, RAM, ROM, PROM, EPROM, hard disk, or any other information processing device, such as a server or computer with which the controller communicates.

[0152] Moreover, the claimed advancements may be provided as utility applications, background daemons, operating system components, or combinations thereof, executing in conjunction with the CPU2000 and an operating system, such as Microsoft's Windows 10, UNIX, Solaris, Linux, Apple's macOS, or other systems known to those of skill in the art.

[0153] To realize the control device, the hardware elements can be realized by various circuit elements known to those skilled in the art. For example, the CPU 2000 can be a Xenon processor or Core processor from Intel, USA, or an Opteron processor from AMD, USA, or other processor types that will be recognized by those skilled in the art. Alternatively, the CPU 2000 can be implemented on an FPGA, an ASIC, or a PLD, or can be implemented using discrete logic circuits that will be recognized by those skilled in the art. Furthermore, the CPU 2000 can be implemented as multiple processors that work cooperatively in parallel to execute the instructions of the process of the present invention described above.

[0154] The control device of FIG. 17 also includes a network controller 2006, such as an Intel Ethernet PRO network interface card from Intel Corporation of America, for connection to a network 2050. As can be appreciated, the network 2050 can be a public network, such as the Internet, a private network, such as a LAN or WAN, or any combination thereof, or can include sub-networks of the PSTN or ISDN. The network 2050 can also be wired, such as an Ethernet network, or wireless, such as a cellular network, including Edge, 3G, and 4G wireless cellular systems. The wireless network can also be WiFi, Bluetooth, or any other known form of wireless communication.

[0155] The control device further includes a display controller 2008, such as an NVIDIA GeForce GTX or Quadro graphics adapter from NVIDIA of America, for connecting to a display device 2010, such as a Hewlett-Packard HPL2445w LCD monitor. A general purpose I / O interface 2012 connects to a keyboard and / or mouse 2014 and a touch panel 2016 on or separate from the display device 2010. The general purpose I / O interface 2012 also connects to a wide variety of peripherals 2018, including any peripherals suitable for electron microscopy.

[0156] Also provided within the control device is a sound controller 2020, such as Creative's Sound Blaster X-Fi Titanium, and connected to a speaker / microphone 2022 to provide sound and / or music.

[0157] A generic storage controller 2024 connects the storage media disk 2004 to a communications bus 2026, which may be ISA, EISA, VESA, PCI, or the like, for interconnecting all of the components of the control device. A description of the general features and functions of the display device 2010, keyboard and / or mouse 2014, as well as the display controller 2008, storage controller 2024, network controller 2006, sound controller 2020, and generic I / O interface 2012 are omitted herein for the sake of brevity, as these features are known.

[0158] The described systems and methods are summarized as follows.

[0159] In one embodiment, a first system for determining a structure and / or elemental composition of a specimen includes an electron source configured to direct an electron beam toward a sample, an electron detector consisting of an array of detector elements, and processing circuitry configured to acquire images from the electron detector using a global shutter readout such that each image from the detector is synchronized with an external event.

[0160] In such an embodiment, the first system further comprises circuitry configured to scan the electron beam over the sample as a probe.

[0161] In the above embodiment, the electron source and the electron detector are part of a transmission electron microscope (TEM).

[0162] In the above embodiment, the electron source and the electron detector are part of a scanning transmission electron microscope (STEM).

[0163] In the above embodiment, the electron source and electron detector are part of a scanning electron microscope (SEM).

[0164] In such an embodiment, the electron detector includes a pixel array configured to directly convert charge from impinging electrons into an electronic signal.

[0165] In the embodiment, the volume ratio of the depletion region associated with the charge integration is 800 times larger than the depletion region associated with the sense node.

[0166] In the above embodiment, the area of ​​the sense node is less than 1.25% of the area of ​​the charge integration region.

[0167] In such an embodiment, the electron detector includes a pixel array configured to detect light produced by the scintillator when electrons strike the electron detector.

[0168] In the embodiment, the electronic detector is a front-illuminated monolithic active pixel array sensor.

[0169] In the embodiment, the electron detector is a back-illuminated monolithic active pixel array sensor.

[0170] In the embodiment, the electronic detector is a hybrid pixel active pixel array sensor.

[0171] In said embodiment, the electronic detector is operated such that the start of integration of the electronic signal and the end of integration of the electronic signal are exactly the same for all pixels.

[0172] In the above embodiment, the random portion of the detector is not read out.

[0173] In the above embodiment, the pseudorandom part of the detector is not read out.

[0174] In the above embodiment, the periodic portion of the detector is not read out.

[0175] In another embodiment, a second system for determining a structure and / or elemental composition of a specimen includes an electron source configured to direct an electron beam toward a sample, an electron detector comprising an array of detector elements, and processing circuitry configured to read out discrete portions from the detector.

[0176] In the above embodiment, the electron source and the electron detector are part of a transmission electron microscope (TEM).

[0177] In the above embodiment, the electron source and the electron detector are part of a scanning transmission electron microscope (STEM).

[0178] In the above embodiment, the electron source and electron detector are part of a scanning electron microscope (SEM).

[0179] In such an embodiment, the electron detector includes a pixel array configured to directly convert charge from impinging electrons into an electronic signal.

[0180] In such an embodiment, the electron detector includes a pixel array configured to detect light produced by the scintillator when electrons strike the electron detector.

[0181] In the embodiment, the electronic detector is a front-illuminated monolithic active pixel array sensor.

[0182] In the embodiment, the electron detector is a back-illuminated monolithic active pixel array sensor.

[0183] In the embodiment, the electronic detector is a hybrid pixel active pixel array sensor.

[0184] In the embodiment, the electronic detector operates with a global shutter readout.

[0185] In the embodiment, the electronic detector operates with rolling shutter readout.

[0186] In the above embodiment, the random portion of the detector is not read out.

[0187] In the above embodiment, the pseudorandom part of the detector is not read out.

[0188] In the above embodiment, the periodic portion of the detector is not read out.

[0189] In the embodiment, discontinuous parts of the detector are not read out and the missing information is restored during the image processing steps.

[0190] In such an embodiment, discontinuous portions of the detector are not read out and the missing information is ignored during the image processing steps.

[0191] In another embodiment, a third system for determining the structure and / or elemental composition of a sample using 4D STEM includes an electron source configured to direct an electron beam to the sample, circuitry configured to scan the electron beam over the sample as a probe, an electron detector consisting of an array of detector elements, and processing circuitry configured to acquire multiple images of bright field disks and surrounding dark field diffraction from the electron detector, calculate radial profiles of the corrected images, normalize the radial profiles by a magnification factor, characterize the shape of the radial profiles at the edge of the bright field disk, and determine the elemental composition based on the radial profiles at the edge of the bright field disk for each image.

[0192] In the embodiment, the detector is operated with a global shutter readout so that each image from the detector is synchronized with the movement of the STEM probe.

[0193] In the embodiment, discontinuous parts of the detector are not read out and the missing information is restored during the image processing steps.

[0194] In such an embodiment, discontinuous portions of the detector are not read out and the missing information is ignored during the image processing steps.

[0195] In said embodiment, the third system further comprises correcting geometric distortion in the image by fitting an ellipse to the bright field disk.

[0196] In the above embodiment, the third system further includes aligning each brightfield disk image of the plurality of images based on a center of gravity of the brightfield disk.

[0197] In the above embodiment, the third system further includes aligning each brightfield disk image of the plurality of images based on a geometric center of an edge of the brightfield disk.

[0198] In the above embodiment, the third system further includes calculating a radial profile of the corrected image.

[0199] In said embodiment, the third system further comprises the step of normalizing the radial profile by the average intensity of a bright field disk.

[0200] In the above embodiment, the third system further includes normalizing the radial profile by the average intensity of the dark field region.

[0201] In said embodiment, the third system further comprises the step of normalizing the radial profile by a combination of the average intensity of both the bright field disk and the dark field region.

[0202] In such an embodiment, the shape of the radial profile is characterized by the gradient of the radial profile at the transition between the bright field region and the dark field region.

[0203] In such an embodiment, the shape of the radial profile is characterized by the similarity between radial profiles of the transitions between bright and dark field regions in different images.

[0204] In the embodiment, the shape of the radial profile is characterized by its similarity to an existing model of the edge shape of the transition between the bright field region and the dark field region.

[0205] Obviously, many modifications and variations are possible in light of the above teachings and, therefore, within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.

[0206] Additionally, embodiments of the present disclosure may be described in the following supplement:

[0207] (1) A method for determining material properties of a sample using 4D STEM, the method comprising: directing an electron beam from an electron source toward the sample; detecting electrons from the electron beam that interact with the sample using an electron detector including an array of detector elements; acquiring from the electron detector a plurality of bright field disks and surrounding dark field diffraction images based on the detected electrons; and determining an elemental composition of the sample based on the acquired plurality of bright field disks and surrounding dark field diffraction images.

[0208] (2) The method of (1), further comprising correcting distortion in the acquired image.

[0209] (3) The method of (2), wherein determining the elemental composition of the sample based on the multiple acquired images of the bright field disk and surrounding dark field diffraction further includes calculating a radial profile of the distortion-corrected acquired images, normalizing the radial profile by magnification, determining a shape of the radial profile at the edge of the bright field disk, and determining the elemental composition based on the shape of the radial profile at the edge of the bright field disk.

[0210] (4) The method of (2) or (3), wherein the distortion in the acquired image is geometric, and correcting the geometric distortion comprises fitting an ellipse to the bright field disk.

[0211] (5) The radial profile is normalized based on the average intensity of a bright-field disk, as in (3).

[0212] (6) Any one of the methods of (1) to (5), wherein the detecting includes detecting electrons using an electron detector in a global shutter readout mode and synchronizing each image acquired from the electron detector with movement of a probe directing the electron beam.

[0213] (7) Any one of the methods of (1) to (6), further comprising aligning each of the images of the multiple bright field disks with respect to each other based on the center of each of the multiple bright field disks.

[0214] (8) The method of (3), wherein the shape of the radial profile corresponds to the gradient of the radial profile of the transition between the bright field disk region and the surrounding dark field diffraction region.

[0215] (9) A method for compressed readout using an electron microscope, the method comprising: generating a row address table including a first set of a reduced number of row addresses and a frame height that is less than a maximum number of row addresses possible within a frame height; transmitting each row address of the reduced number of row addresses in the row address table to a control device; the control device configured to control an electron source to direct an electron beam toward a sample to read out charge from an electron detector including an array of detector elements, the electron detector configured to detect electrons in the electron beam; controlling the control device to read out according to the transmitted row addresses; determining whether a last row address in the row address table has been read out; and if it is determined that the last row address in the row address table has not been read out, controlling the control device to increment a pointer to a next row address in the row address table and read out according to the next row address; and repeating the control and determination until the last row address in the row address table has been scanned.

[0216] (10) The method of (9), wherein the reduced number of row addresses in the row address table is reduced from the maximum number of possible row addresses within the frame height by omitting random selection of row addresses.

[0217] (11) Either method (9) or (10), in which the reduced number of row addresses in the row address table is reduced from the maximum number of row addresses possible within the frame height by omitting periodic selection of row addresses.

[0218] (12) Any one of the methods of (9) to (11), wherein generating the row address table includes generating a second set of reduced number row addresses, the second set of reduced number row addresses being included in the row address table following the first set of reduced number row addresses, and at least one row address in the second set of reduced number row addresses being different from the row addresses in the first set of reduced number row addresses.

[0219] (13) Any one of the methods of (9) to (12), further comprising obtaining a first image from the electronic detector based on the reduced number of row addresses read out from the electronic detector, and generating a second image that is an expanded reconstruction based on the first image.

[0220] (14) A system for determining a material property of a sample, the system comprising: an electron detector including an array of detector elements configured to convert charge from impinging electrons into an electronic signal; and processing circuitry configured to read out the electronic signal from each detector element of the detector array in a global shutter mode.

[0221] (15) The system of (14), wherein the electron detector is a direct electron detector configured to directly convert charge from impinging electrons into an electronic signal.

[0222] (16) A system as in (15), in which the volume ratio of the first depletion region associated with the charge integral is greater than 200 times the volume ratio of the second depletion region associated with the sense node.

[0223] (17) A system according to (15), in which the area of ​​the sense node is less than 5% of the area of ​​the charge integration region.

[0224] (18) Any one of the systems (14) to (17), wherein the detector array is positioned so that a front surface of the detector array detects impinging electrons.

[0225] (19) Any one of the systems (14) to (18), wherein the electronic detector is a hybrid pixel active pixel sensor.

[0226] (20) Any one of the systems (14) to (19), wherein the processing circuitry is further configured to exclude a predetermined number of row addresses while reading out the electronic signals from each detection element of the detection element array in global shutter mode.

[0227] Thus, the foregoing discussion discloses and describes only a few exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. Accordingly, the disclosure of the present invention is intended to be illustrative, and not limiting, of the scope of the present invention and other claims. The present disclosure, including any readily identifiable variations of the teachings herein, defines in part the scope of the preceding claim terms, so as not to indulge the subject matter of the present invention to the general public.

Claims

1. 1. A method for determining material properties of a sample using 4D STEM, comprising: Directing an electron beam from an electron source onto the sample; detecting electrons of the electron beam interacting with the sample using an electron detector including an array of detector elements; obtaining a plurality of bright field disks and surrounding dark field diffraction images based on the detected electrons from the electron detector; determining an elemental composition of the sample based on the acquired images of the bright field disk and the surrounding dark field diffraction; the detecting includes detecting the electrons using the electron detector in a global shutter readout mode and synchronizing each of the images acquired from the electron detector with a movement of a probe directing the electron beam; correcting distortions in the acquired image; Determining the elemental composition of the sample based on the acquired images of the bright field disk and the surrounding dark field diffraction includes: Calculating a radial profile of the distortion corrected acquired image; Normalizing the radial profile by a scale factor; determining a shape of the radial profile of edges of the plurality of bright field disks; determining the elemental composition based on the shape of the radial profile at the edges of the plurality of bright field disks.

2. The method of claim 1 , wherein the distortion in the acquired image is geometric, and correcting the geometric distortion comprises fitting an ellipse to the plurality of bright field disks.

3. The method of claim 1 , wherein the radial profile is normalized based on an average intensity of the plurality of bright field disks.

4. aligning the images of the bright field disks with respect to one another based on respective centers of the bright field disks; The method of claim 1 further comprising:

5. The method of claim 1 , wherein the shape of the radial profile corresponds to a gradient of the radial profile of a transition between the multiple bright field disk regions and the surrounding dark field diffraction regions.

Citation Information

Patent Citations

  • Focusing ion beam processing monitoring device, sample processing method, and sample monitoring method

    JP2007214088A

  • Scanning transmission electron microscope, and image generation method

    JP2018088321A

  • Optical axis adjustment method and electron beam inspection device

    JP2018142472A

  • Distortion measurement method of electron microscopic image, electron microscope, sample for measuring distortion, and method of manufacturing sample for measuring distortion

    JP2018200810A

  • Image based systems for detecting information on moving objects

    US20140078314A1