An automated and robust method for recording nanometer-resolution 3D image data from serial microsections of life science samples using electron microscopy

Abstract

To provide a method for accurately imaging a selected portion of a sample section.SOLUTION: An aligning method of a specimen image of a specimen section positioned on a substrate includes obtaining an optical or SEM image of the substrate, locating the optical or SEM image of each specimen section, and perform alignment. The specimen section is then imaged in a SEM to obtain a preview image and a region of interest (ROI) in at least one of the preview images is selected. The preview image is processed such that at least a portion of the preview image close to the ROI is aligned with the preview image. On the basis of the alignment of the preview image, the final SEM image of the selected specimen section is obtained such that a set of aligned images in 3D can be used. The image alignment can use cross-correlation with fixed or variable criteria that can be updated as the specimen section image is processed.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 868,617, filed June 28, 2019, which is incorporated by reference in its entirety.

[0002] The present disclosure relates to obtaining precisely matched sub-areas of multiple sections using a scanning electron microscope. [Background technology]

[0003] Ultrafine structural information of tissue samples is becoming increasingly important in life science research. Although scanning electron microscopes (SEMs) can obtain high-resolution images, tissue samples cannot efficiently generate the secondary or backscattered electrons required for SEM analysis. Therefore, samples are stained with heavy metals (osmium, lead, etc.) and then embedded in resin boxes for trimming and sectioning. These resulting blocks are approximately 1 mm in size. 3 . Imaging in an SEM is limited to the block surface because electrons do not efficiently penetrate deeper than about 30 nm into the block. To generate 3D image data, the block must be cut into serial sections or a thin layer must be removed from the block surface during imaging. Approaches based on layer removal, such as serial block face imaging or FIB-SEM dual beam imaging, destroy the sample and require the user to acquire large-scale data during imaging, when the sections cannot be re-imaged. Cutting the sample into sections used in so-called array tomography, where each section can be re-imaged as needed and the user does not need to be involved in sample destruction. Commercially available ultramicrotomes can reproducibly cut 50 nm sections, and then 100-300 serial sections are cut, typically on a 10 cm wafer or up to 10 cm. 2 cm 2 The electrode is placed on a conductive support of a metal plate having an area of ​​10 mm.

[0004] It is rarely necessary to acquire high-resolution image data from the entire tissue block. Also, a 1 mm section recorded at a resolution of 10 × 10 × 50 nm pixel resolution is sufficient. 3 A block represents 3 petabytes of data, which would take 105 days to record with a beam dwell time of 3 μs, which is not feasible. The volume of interest is typically the size of one or a few biological cells, i.e., 30 3 μm 3 ~100 3 μm 3 The main obstacle in SEM imaging of serial sections is the need to obtain the same 30 μm resolution for each of the hundreds of serial sections scattered across the surface of the sample support. 2 μm 2 ~100 2 μm 2 While the SEM technician could navigate in this manner, it could take a significant amount of time, from hours to days, to identify the appropriate section location and align these portions of the section. For these and other reasons, an alternative approach is needed. Summary of the Invention

[0005] Disclosed are methods and apparatus that enable imaging of selected portions of a specimen section with high accuracy, i.e., with minimal section-to-section variation in placement of the imaging area. In some examples, the preview images are processed for registration based on at least one feature in one or more of the preview images. Alternatively, each preview image is processed for registration based on a search template selected from the set of preview images. In a typical example, a set of specimen images associated with an ROI is obtained, the specimen images having a higher resolution than the resolution associated with the preview image. In some examples, each preview image is processed for registration based on a correlation with a search template image selected from the set of preview images. In a particular example, the set of preview images includes N preview images 0, ..., N, where N is an integer, and at least one preview image is processed for registration based on a search template image selected from the set of preview images. For example, the i-th preview image is registered by comparison with the (i-1)-th preview image, where i is an integer greater than 1 and less than N. In some embodiments, each of the preview images of the set of preview images is registered by aligning the preview images or storing an image transform associated with the alignment. In a further example, the preview images are associated with a first resolution and a set of ROI images having a second resolution are obtained based on the registered preview images, the second resolution being higher than the first resolution.

[0006] In another alternative, an image including image areas associated with a plurality of specimen slices is obtained and processed to identify image areas associated with the specimen slices. The positions of the slices are established based on the identified image areas, and each of the preview images is associated with a respective specimen slice. In a representative example, stage coordinates associated with the alignment of the image areas associated with the specimen slices are obtained and stored. In a representative example, an ROI image is obtained for each of the selected slices, and a three-dimensional reconstruction of at least a portion of the ROI is generated based on the alignment of the preview images or based on a high-resolution image aligned based on the alignment of the preview images.

[0007] The system includes an imaging device arranged to obtain an overview image of a substrate including a plurality of specimen sections. A first image processor is coupled to receive the overview image and locate image portions corresponding to the plurality of specimen sections. A charged particle beam (CPB) imaging system is configured to generate preview images corresponding to selected portions of each specimen section. A second image processor is coupled to receive the preview images and determine an alignment of the preview images. In some examples, the CPB imaging system is configured to generate ROI images corresponding to each specimen section and align the ROI images based on the alignment of the preview images. In representative examples, the first image processor and the second image processor are the same image processor. In other examples, the overview image has a first resolution, the preview image has a second resolution higher than the first resolution, and the ROI image has a third resolution higher than the second resolution. According to some examples, the first image processor is coupled to locate image portions associated with the plurality of specimen sections based on a correlation with a section template. In further examples, the second image processor is coupled to align the preview image based on correlation with one or more search templates or based on feature identification. In some examples, the first image processor is coupled to determine a substrate stage position corresponding to the alignment of the image of the specimen section, and the second image processor is coupled to determine a substrate stage position corresponding to the alignment of the preview image.

[0008] The method includes using a processor to identify a plurality of specimen slices of the 3D specimen in an overview image of the plurality of slices of the 3D specimen based on a slice template, the overview image being an optical image associated with a first image resolution. The images of the identified plurality of specimen slices are registered, and refinement regions including a region of interest in at least a selected set of the images of the identified slices are selected. Preview images including each of the refinement regions are obtained, the preview images being electron beam based images having a second image resolution higher than the first image resolution. The preview images are registered relative to one another using the feature identification, and an electron beam based image associated with a ROI is obtained for each of the preview images, the electron beam based image having a third image resolution higher than the second image resolution. For each registered preview image, a registered electron beam based image associated with the ROI is stored.

[0009] These and other features of the disclosed technology are illustrated below with reference to the accompanying drawings. [Brief description of the drawings]

[0010] [Figure 1A] 1 shows an exemplary method for alignment and registration of images from a sectioned sample. [Figure 1B] 1 shows an exemplary method for alignment and registration of images from a sectioned sample. [Figure 2A] 13 shows images related to alignment of slice images. [Figure 2B] 13 shows images related to alignment of slice images. [Figure 2C] 13 shows images related to alignment of slice images. [Figure 2D] 13 shows images related to alignment of slice images. [Figure 2E] 13 shows images related to alignment of slice images. [Figure 2F] 13 shows images related to alignment of slice images. [Figure 2G] 13 shows images related to alignment of slice images. [Figure 2H] 13 shows images related to alignment of slice images. [Figure 2I] 13 shows images related to alignment of slice images. [Diagram 3] 1 shows an exemplary imaging device that produces images in registration. [Figure 4] 1 illustrates a representative computing environment for controlling image acquisition and processing. [Diagram 5] Illustrates a method for aligning images. [Figure 6A] A representative method for aligning slice images is shown. [Figure 6B] A representative method for aligning slice images is shown. [Figure 6C] A representative method for aligning slice images is shown. [Figure 6D] A representative method for aligning slice images is shown. [Figure 6E] A representative method for aligning slice images is shown. [Figure 7] A representative method for acquiring a series of images for 3D tomography is shown. [Figure 8A] Section images with and without distortion are shown. [Figure 8B] Section images with and without distortion are shown. [Figure 8C] Sections positioned on the tape are shown. [Figure 8D] 1 illustrates slice image artifacts associated with slice images acquired at different slice positions within the imaging field of view. [Figure 8E] 1 illustrates slice image artifacts associated with slice images acquired at different slice positions within the imaging field of view. [Figure 8F] We show that aligning the sections within the imaging field reduces or eliminates section image artifacts associated with stitching section images. [Figure 8G]We show that aligning the sections within the imaging field reduces or eliminates section image artifacts associated with stitching section images. [Figure 9A] A preview image with a ribbon-like specimen section is used to demonstrate alignment of images within an image stack. [Figure 9B] A preview image with a ribbon-like specimen section is used to demonstrate alignment of images within an image stack. [Figure 9C] A preview image with a ribbon-like specimen section is used to demonstrate alignment of images within an image stack. [Figure 9D] A preview image with a ribbon-like specimen section is used to demonstrate alignment of images within an image stack. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Definitions and Terminology As used herein, "image" refers to a viewable image presented on a display or otherwise made viewable by a user, and to stored representations adapted to generate such viewable images. Examples of such representations include files in formats such as .jpg, .tiff, .bmp, etc., stored on a computer-readable medium such as a hard disk drive, memory, or otherwise stored. Images may store intensity or other values ​​as a function of coordinates, such as intensity I(x,y), where x,y are Cartesian coordinates. Other representations are possible, such as three-dimensional representations using Cartesian, polar, or other coordinates. For convenient description, the methods are described as a sequence of certain steps, although in some cases these steps may be performed in a different order and one or more steps may be in progress simultaneously. In some cases, images or image portions are referred to as being registered or in registration. As used herein, these terms refer to images of specimen sections that have been processed by rotation and / or translation to overlap to correspond to the position and orientation of the specimen prior to sectioning. Alternatively, these terms refer to images that have been processed to identify the location and orientation of the specimen prior to sectioning, by rotation, translation, or other processing that allows for the generation of images with the specimen's location and orientation prior to sectioning. For example, image coordinates can be updated so that all images are specified in a common coordinate system, or each image can be defined with respect to its own or other coordinate system, but offsets and / or rotations can be used to overlay or otherwise align the images as needed. In either case, the images can then be used to determine the structure of the specimen through a stack of section images. The alignment can be used to determine the stage coordinates of a light or CPB microscope for acquiring a suitable image.

[0012] In some examples, correlation with one or more reference images or templates is used to determine image alignment. Fixed or variable references or templates can be used. Typically, precise alignment of layer images uses a reference image that may vary with the section images in the stack. For example, an image processed for alignment with respect to the reference can be used as a reference for aligning subsequent images. The reference image can change at least for the section images in the stack, or it can change every second, third, fourth, fifth, or other interval. Features can be tracked from layer to layer, or correlation between layers can be calculated and the maximum of the correlation used to indicate alignment.

[0013] Overview of the Example An example is described for processing sections of a specimen obtained using a microtome. A number of sections are located on a substrate and a first alignment procedure ("coarse alignment") is used to position the sections relative to one another based on a comparison or correlation with a section template, typically selected by the user among the section images contained in the overview image of the substrate. In a second alignment procedure ("fine alignment"), so-called preview images of some or all sections are obtained. From the selected sections, an ROI template is selected and the first preview image is aligned based on a comparison or correlation with the ROI template. Based on the aligned first preview image, the ROI template is updated and the second preview image is aligned based on a comparison or correlation with the updated ROI template. This process is repeated for all preview images of interest. Then, an ROI is selected by the user and aligned high-resolution images can be obtained from some or all sections. In the example considered below, the overview image is an optical image and the subsequent images (preview image and high-resolution image) are electron beam images. However, optical images or charged particle beam images can be used, such as those produced by electron microscopes, optical microscopes, scanning light microscopes, ion beam images, and the like.

[0014] Example 1. Tomography images from specimen sections The disclosed methods and apparatus can be used in 3D and other imaging of specimens, such as biological specimens. An exemplary method 100 is shown in FIG. 1. At 102, a specimen block or other 3D specimen is obtained. The specimen block is then sectioned at 104, for example using a high-precision microtome, with the sections typically arranged on a substrate in the order in which they were removed from the specimen block. A silicon wafer is a convenient substrate. A set of one or more substrates holding multiple specimen sections is referred to herein as an "array tomography sample" or simply "sample." The sections can be arbitrarily arranged with appropriate tracking of the sections to preserve the order of the sections if desired, although sequential ordering is typically more convenient. For example, the order of the sections can be stored at 105. 3 Sections obtained from a specimen block generally do not fit onto a single substrate, and 10-100 wafers may be required. In most instances, the sections are arranged in rows extending along parallel axes (e.g., from left to right), with each subsequent row beginning at a new row position proximate to the position of the initial section of the preceding row. Alternatively, a new row may be started by placing a section near the last section of the previous row, returning to the leftmost position upon completion of the row. Although not discussed in detail, the specimen block and sections may be stained or labeled for light or electron beam microscopy, as desired. For example, fluorescent immunolabels and heavy metals may be used for light and electron beam microscopy, respectively. In some cases, the sections are placed on tape, and the tape is then placed on the substrate. The substrate is coupled to a stage for positioning for light and electron beam imaging, and feature and image positions may be specified based on stage coordinates and rotation.

[0015] The section (i.e. the sample) as placed on the substrate is imaged with an optical imaging device 108 such as a camera to obtain one or more overview images. For convenience in what follows it is assumed that only a single substrate and a single overview image are required. The overview image is processed at 110 to identify the section and obtain the associated position, typically as xy coordinates in a coordinate system with x and y axes in the plane of the substrate surface holding the section. A stage position (and orientation) is assigned to each section and recorded at 111. The sections can be detected using a user-defined template and the overview image using, for example, cross-correlation of the optical image with the template. The template is typically selected among the section images of the overview image and is referred to herein as a "section template". Multiple portions of the one or more overview images can be processed with cross-correlation in parallel so that the sections can be more quickly identified and located. An image resolution of about 2 μm / pixel is used so that the image or a portion thereof can be used in the correlation operation, although with high resolution images the differences between successive images may be too large for cross-correlation to work. An image of a particular slice can be identified and used as a slice template for all slice locations.

[0016] With the sections located and ordered, a section preview image (typically using the electron beam) is obtained at 112. The preview image is associated with the portion of the section that contains the region of interest (ROI). In some examples, the preview image is obtained based on a user outline provided on the section image, and a graphical user interface can be provided for such selection. Preview images typically cover the entire section and are 1%, 5%, or 10% of the size defined in the section template, although other sizes can be used. These preview images are typically obtained at a better resolution (e.g., 1 μm / pixel or better) than the resolution used at the section locations. Preview images can exhibit variable artifacts such as distortions within the image, variable magnification between images, etc. As a result, stage positions typically do not provide perfect or satisfactory stack alignment of sections of ROIs at various locations in the preview image. However, a preferred location for each ROI can be obtained using feature-based image alignment or image correlation, using a search template that can be updated during processing to allow image portions within each ROI to be aligned with image portions adjacent to it. The search template is typically selected as at least a portion of the first preview image. At 114, the preview images are aligned by tracking image features from one preview image to the next using the search template. In some examples, the features used for tracking are updated after one or more preview images are aligned to accommodate specimen variability, and the search template is updated after each preview image is processed. In some examples, the selected preview image is used as a search template for cross-correlation with one or more other preview images. During processing, the search template can be updated as needed. Each ROI typically requires independent feature-based alignment or other alignment due to image distortion and other image artifacts, but ROIs that are sufficiently close may not.The acceptable degree of closeness may be a function of the size and proximity of the image artifacts, and it may be more convenient to align each ROI using a dedicated feature-based alignment for each ROI. At 116, the preview image alignment and registration values ​​may be obtained and, at 115, typically stored in a computer-readable medium.

[0017] Once the image stack is aligned, a high resolution image (e.g., 2 nm / pixel) can be acquired at 118 and used for 3D reconstruction at 120. In some examples, the alignment of the preview images can be repeated, typically by acquiring and processing additional preview images having a higher resolution than the first preview image. If additional ROIs are to be investigated, the process returns to 112, where suitable preview images associated with the additional ROIs are obtained and processed. In this example, the slices have been previously located and the associated method steps are not necessary.

[0018] Example 2. Previewing image alignment FIG. 1B illustrates an exemplary method 150 of image stack registration using preview images or other image portions. For ease of explanation, FIG. 1B is discussed with reference to an image stack of slices 0, 1, 2, ..., N, where N is an integer and slice 0 is the top slice. At 152, a set of preview images is received and a search template is selected, typically the preview image of the 0th slice (or other slice). At 156, a preview image of the i-th layer is selected and compared and registered, typically using cross-correlation, at 158. At 160, the registration coordinates are stored, typically as stage coordinates for subsequent registration. At 162, it is determined whether to register additional preview images and, if so, at 154, an updated search template is selected. In some examples, the updated search template is the i-th image used in the previous registration, while in other examples, the initial reference image (0th image) is used. In other examples, a different updated search template may be selected after processing 2, 5, 10, 20, 50, or 100 images, or the most recently used preview image may be selected. Varying the updated search template through the section stack may maintain registration even in the presence of image features that change across the stack. In some examples, only the initially selected updated search template is appropriate and any preview image may be used. More generally, continually updating the updated search template from section to section allows registration across hundreds of sections, even in the case of progressively changing biological structures. Once registration is complete, the ROI portion of the preview image may be used to obtain a final high-resolution image and establish a 3D image of the volumetric region of interest.

[0019] The alignment of the stack can proceed from within the stack and does not have to start with the top or bottom slice. For example, the kth preview image can be selected as the updated search template, and the (k-1)th and (k+1)th preview images can be aligned to update the search template as desired. Preview images can be processed sequentially, or multiple preview images can be processed in parallel as desired.

[0020] The specimen portion of interest typically extends only through a selected portion of the specimen block. In such instances, images corresponding to all sections and all areas of the sections are not necessary. A user can conveniently select any section and area of ​​interest using a graphical user interface.

[0021] Example 3. Representative specimen treatment 2A-2I illustrate specimen processing and alignment of ROI images. FIG. 2A shows an image 200 of a substrate 202 supporting a series of tape strips, such as representative tape strip 204, holding sample sections, such as representative sample section 206. Although twelve tape strips are shown, more or fewer may be used, and the sample sections may be placed directly on the substrate 202. Multiple such substrates may be required to hold all sections of the specimen. Image 200 is an overview image, and a particular image portion 208 of the overview image 200 is selected to include an image 210 of a single section. The image portion 208 is typically selected for use as a slice template in identifying and locating other sections using correlation of the slice template with the overview image 200, and then the relative displacement corresponding to a large value of the correlation coefficient is identified and coordinates obtained such that the location of the slice image is established.

[0022] FIG. 2B shows an image portion 212 including images of a number of slices, where slices 220, 221 are identified as shown by frames such as frame 209, further shown in FIG. 2C. The images of each of these slices are shown with associated coordinate axes of a two-dimensional xy coordinate system. For example, slices 220, 221 correspond to respective coordinate axes 230, 231, as shown in FIGS. 2D-2E. In this example, slice dimensions are approximately 1 mm×0.5 mm, and the location of imaging regions 240, 241 relative to respective coordinate axes 230, 231 are shown along with rotation angle r. Imaging regions are generally only required at selected slices, which may be conveniently selected with a trace 250 extending through all slices of interest, as shown in FIGS. 2F-2G. Selection of slices may be accomplished in a variety of ways, such as by drawing a trace 250 on the displayed image of the substrate, or by using a computer-based pointing device to draw the slices. A cursor 256 may be used in establishing the trace 250 and may be manipulated via computer-executable instructions for a mouse, trackpad, keyboard, or other device.

[0023] The selected sections are identified and are at specified locations relative to each other, but generally are not well aligned to the Regions of Interest (ROIs). Alignment and registration is limited by magnification errors, rotation errors, non-linear distortions of tiled SEM overview images acquired with a wide field of view, and inconsistencies at the boundaries of the sections, which may result in the sections not being well aligned and positioned in the stack. As shown in Figures 2H-2I, a portion 260 of the section 220 is selected and a preview image 262 of area 261 is acquired. The preview image 262 and area 261 are selected to include an ROI that includes the specimen features of interest. The preview image 262 is typically a higher resolution image than a previous image, such as an overview image. Preview images including ROIs for other selected sections may be similarly acquired, and the preview images are then aligned using correlation or other processes as discussed above. In a preview image-based alignment, an initial or previously aligned preview image or portion thereof may be used as a search template, and the search template is updated after processing each preview image. The preview images from each slice may be aligned; typically, only the preview images shown in trace 250 are aligned, although in some instances, hundreds of such preview images are selected.

[0024] After alignment of the preview image, the preview image can be provided for use in generating a 3D image of the ROI. Alternatively, with this alignment, higher resolution images can be acquired and the substrate stage can be used to properly position the section. If necessary, these additional images can also be aligned. In either case, the resulting image stack allows for 3D reconstruction with minimal operator intervention.

[0025] In some examples, the user specifies an area surrounding the ROI, and both the linear angles (e.g., xy coordinates) and rotation angles are adjusted to match the search template. For example, an ROI is selected from a first slice as the search template, and a corresponding portion of a second slice is aligned to match, applying appropriate translations and rotations. With the first and second slices aligned, image portions around the ROI in the image of the second slice are selected for use in processing the third slice. This process continues until all slices of interest have been processed. In some examples, the images are not adjusted, but the preferred xy offsets and rotation angles are saved for use in subsequent image processing. As discussed above, other areas of the slice require different offsets and rotations, and images can be acquired, processed, aligned, and stored for multiple areas.

[0026] Example 4. Imaging system 3, imaging system 300 includes a system controller 302 coupled to an ion beam source 304, an electron beam source 306 producing an ion beam 305, and an electron beam 307, respectively. Respective scanners 312, 314 are positioned to direct a scanned ion beam 313 and a scanned electron beam 315, respectively, toward a specimen 320. In some applications, images are obtained based on the scanned electron beam 315, and the scanned ion beam 313 is used for specimen modification. However, images can be obtained with either one of the scanned ion beam 313 and the scanned electron beam 315, or both. In some cases, the imaging system includes only one of the electron beam source and the ion beam source. For many biological specimens, only the electron beam is required.

[0027] The specimen 320 is fixed to a stage 322 that is coupled to a stage controller 324, which is coupled to the system controller 302. The stage 322 can typically provide one or more translations, rotations, or tilts as directed by the system controller 302. A beam 326 responsive to the scanned ion beam 313 or scanned electron beam 315 is directed to an electron or ion detector 328 that is coupled to system electronics 330, which may include one or more analog-to-digital converters (ADCs), digital-to-analog converters (DACs), amplifiers, and buffers for control of the detector 328 and processing (amplification, digitization, buffering) signals associated with the detector 328. In other examples, photon detectors are used that produce electrical signals that are further processed by the system electronics. In most practical examples, at least one ADC is used to produce a digitized detector signal that can be stored in one or more tangible computer-readable media, such as an image (shown as image storage 332). In other examples, the image store is remotely located via a communications connection, such as a wired or wireless network connection. The beam 326 may be a scanned ion beam 313, a scanned electron beam 315, secondary electrons, ions, or scattered portions of neutral atoms. An optical imaging element 351, such as a camera, is coupled to produce an image of the specimen 320, for example to produce a substrate image representative of a plurality of substrate slices. As described above, such images may be processed to identify, locate, and align each of the slices for further (typically higher resolution) imaging using a Charged Particle Beam (CPB).

[0028] The system controller 302 is coupled to a memory 335 storing executable instructions for image processing such as slice identification 336, correlation and feature alignment 340, selection of ROI and preview image areas 338, storing and retrieving overview images 339, and selecting and updating search templates 341 to provide a GUI 342 with various functions including selecting the slice to be processed and defining a visible trace representing the slice of interest. Images (both CPB and optical) may be stored in the memory portion 332. Stage coordinates (including rotation) may likewise be stored in the memory portion 332. The system controller 302 sets image acquisition parameters and is in communication with the stage controller 324. Specimen images (preview images, slice images, board images, overview images, etc.) may be presented on a display 352 and system control and imaging parameters may be manifested using internally stored values ​​from the memory 335 or provided by a user with one or more user input devices 350.

[0029] It will be appreciated that the layout of Figure 3 is for convenience of illustration and does not show the actual alignment of the various beam sources, optical camera 352, and CPB detector. Although a dual beam (ion / electron) system is shown, either or both can be used, and in many applications such as electron microscopes, only the electron beam is used for imaging.

[0030] Example 5. Representative Computing Environment FIG. 4 and the following discussion are intended to provide a brief, general description of an exemplary computing environment in which the disclosed technology can be implemented. In particular, some or all parts of this computing environment can be used in the methods and apparatus described above to control beam scanning and image processing, for example, to identify and register section images, preview images, and image storage. Although not required, the disclosed technology is described in the general context of computer-executable instructions, such as program modules, executed by a personal computer (PC). Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The disclosed technology can also be implemented in other computer system configurations, including handheld devices, tablets, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The disclosed technology can also be implemented in distributed computing environments in which tasks are performed by remote processing devices that are linked via a communications network. In distributed computing environments, program modules can be located in both local and remote memory storage devices. In some cases, such processing is effected in a SEM. The system of the present disclosure may serve to control image acquisition and provide a user interface, as well as act as an image processor.

[0031] 4, an exemplary system implementing the disclosed technology includes a general-purpose computing device in the form of an exemplary conventional PC 400, including one or more processing units 402, a system memory 404, and a system bus 406 coupling the one or more processing units 402 to various system components including the system memory 404. The system bus 406 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any bus architecture. The exemplary system memory 404 includes a read only memory (ROM) 408 and a random access memory (RAM) 410. A basic input / output system (BIOS) 412 containing the basic routines that help to transfer information between elements within the PC 400 is stored on the ROM 408.

[0032] The exemplary PC 400 further includes one or more storage devices 430, such as a hard disk drive for reading from and writing to a hard disk, a magnetic disk drive for reading from and writing to a removable magnetic disk, and an optical disk drive for reading from and writing to a removable optical disk (such as a CD-ROM or other optical media). Such storage devices may be connected to the system bus 406 by a hard disk drive interface, a magnetic disk drive interface, and an optical drive interface, respectively. The drives and associated computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the PC 400. Other computer-readable media can store data accessible by a PC, such as magnetic cassettes, flash memory cards, digital video disks, CDs, DVDs, RAM, ROM, and the like, and can be used in the exemplary operating environment.

[0033] A number of program modules can be stored in the storage device 430, including an operating system, one or more application programs, other program modules, and program data. A user can enter commands and information into the PC 400 via one or more input devices 440, such as a keyboard and a pointing device such as a mouse. For example, a user can enter a command to initiate image capture or select, for example, whether to use optical flow or image difference to locate the charged area. Other input devices can include digital cameras, microphones, joysticks, game pads, satellite receivers, scanners, and the like. These and other input devices are often connected to the one or more processing units 402 through a serial port interface coupled to the system bus 406, but may also be connected by other interfaces, such as a parallel port, game port, Universal Serial Bus (USB), or a wired or wireless network connection. A monitor 446 or other type of display device is also connected to the system bus 406 via an interface, such as a video adapter, and may display, for example, one or more slice images (i.e., images used in identifying and locating slices), preview images, ROI images, or raw images, or processed images, such as aligned images or images with representations of translation and rotation required for alignment. The monitor 446 may also be used to select slices to be processed, or to select alignment of particular images and alignment procedures, such as correlation, feature identification, and preview area selection or other image selection. Other peripheral output devices, such as speakers and a printer (not shown), may also be included.

[0034] The PC 400 can operate in a network environment using logical connections to one or more remote computers, such as a remote computer 460. In some examples, one or more network or communication connections 450 are included. The remote computer 460 may be another PC, a server, a router, a network PC, or a peer device or other common network node, and typically includes many or all of the elements described above with respect to the PC 400, although only a storage device 462 is illustrated in FIG. 4. The personal computer 400 and / or the remote computer 460 can be connected to logical local area networks (LANs) and wide area networks (WANs). Such networking environments are common in offices, enterprise-wide computer networks, intranets, and the Internet. In some examples, the registered stack of images is sent to a remote system for 3D image reconstruction or other processing.

[0035] 4, memory 490 (or a portion of this or other memory) stores processor executable instructions for image acquisition to establish shots, frame times, beam currents, scan speeds, and image processing. Additionally, memory 490 includes processor executable instructions for setting cross-correlation, image registration such as image rotation and translation, selecting reference images and ROIs, recording stage coordinates for registration. In some examples, the processor executable instructions produce display images showing slice identification, processing preview images, and acquiring additional images.

[0036] Example 6. Representative method for section image alignment 5 illustrates a method 500 for producing a set of registered images of selected portions of multiple specimen sections. At 502, a substrate image (typically an optical image) including multiple section images is obtained and displayed, and at 502, a localized section image (typically an SEM image) is obtained and displayed. At 506, a preview image is obtained, and at 508, a refinement region is selected from one or more section images. At 510, the image locations are refined, and at 512, the registered images are stored or output. Alternatively, for each image, the corresponding translations and rotations may be stored, and unregistered images may be output along with these translations and rotations. In some cases, the method 500 requires little user input and proceeds after selection of the ROI.

[0037] Example 7. Sample preparation and tomography processing 6A-6E illustrate an exemplary method 600. Referring to FIG. 6A, an animal or tissue biopsy is taken at 602 and the sample is prepared for imaging at 604. The biopsy tissue is 2 mm 3 The block is then trimmed into blocks of less than 0.5 mm in diameter, which are then further subjected to some or all of the following: chemical fixation, heavy metal staining, and resin impregnation and curing, and then trimmed for serial sectioning. A typical resin block has a frontal surface area of ​​0.5–2 mm 2 At 606, the resin block is serially sectioned using an ultramicrotome into 40-100 nm thick sections. The sections are collected on a substrate such as tape, a glass plate, or a wafer. If tape is used, one or several lengths of tape are glued onto the wafer or onto a metal plate. This collection of sections on a substrate is called an "array tomography sample" or simply a "sample." In some cases, the collection may span multiple substrates, all of which may be included in the array tomography sample. The sample is then prepared at 610 for unattended data acquisition as discussed further below in connection with FIGS. 6B-6D, and at 670, a high-resolution image is obtained.

[0038] A method 620 of preparing data acquisition, such as that used in 610 above, is shown in FIG. 6B. At 622, the sample is placed on the microscope specimen stage, and at 624, the coordinates of all slices on the sample are determined. Determining slice coordinates is discussed further below in connection with FIG. 6C. After slice coordinates are obtained, coordinates of all or selected slices are obtained at ROI coordinates 626. At 628, an imaging region is created for the selected slice. In some cases, an optical image of the entire array tomography sample is recorded in the SEM. This image shows all or most of the slice at a coarse resolution and can be used to define the area from which an overview image is to be acquired.

[0039] Referring to FIG. 6C, a method 630 for determining slice coordinates includes obtaining an optical image of the sample at 632. This image can be used to define the sample area from which an overview image is obtained at 636. The overview image represents multiple slices and can be a single image or an image mosaic. The resolution of the overview image is equally coarse, i.e. 1-2 μm pixel size, so that the entire substrate can be imaged in a reasonable time. There need not be any correlation between the number of images and the number of slices. One image may represent several slices or multiple images may be required to represent just one slice. This depends on the maximum field of view of the microscope and the size of the slices. In some cases, the overview image is obtained by various means other than SEM. After importing such an overview image, it needs to be aligned so that the slice positions in the imported image correspond to the stage coordinates of the same slice in the array tomography sample. This can be accomplished by manual alignment of two or three landmarks found in both the SEM image and the imported image.

[0040] The overview image is used as follows: At 636, a user-selected area (usually a rectangle) is chosen from the overview image. The selected area is copied from the overview image to serve as a section template. In an automatic procedure, at 638, the section template is correlated with the overview image to determine the section locations in the overview image. At 640, the matching locations in the overview image are translated to positions in the stage coordinate system, i.e., the stage position of each section is stored. Moving the stage to one of the stored positions centers each section under the microscope imaging system (the "polar section"). Any sections not found by the automatic procedure can be added to the overview image by the user marking them. Any sections that are incorrectly recognized (i.e., non-sections) can be annotated as false positives and removed from the list of found sections.

[0041] FIG. 6D illustrates a method 650 for determining coordinates of an ROI across all sections. At 652, a section preview image is acquired. The section preview image typically has the same or higher resolution as the overview image. In some examples, a pixel size of 200-800 nm is used. At 654, a section preview image is selected, and at 656, it is determined whether the selected section preview image is of a first section. If so, the ROI is annotated in the first section preview image, for example, by outlining it on a display device using a computer pointing device, and stored as a search template at 658. At 660, a match with the search template in the next section preview image is identified, and at 662, the associated positions and angles are stored and / or translated to stage coordinates. If at 664, it is determined that more sections should be processed, at 654, the next section preview image is selected, and at 666, the search template is updated to the match location in the section preview image identified so far. As each slice preview image is processed to match the search template, the matching area of ​​this slice preview image is set to the search template, thus narrowing the search for matching areas step by step.

[0042] Once all sections have been processed for alignment as discussed above, images are acquired using method 680 shown in FIG. 6E. In 682, an imaged area is defined by the user in one of the sections, and a corresponding area is selected in the other sections. In 684, an image of the imaged area is acquired. In a typical example, the pixel resolution is 5-50 nm and the field of view is 30 nm. 2 μm 2 ~100 2 μm 2 Since the stage coordinates for each of the slices are obtained, the method 680 can be executed by the processor without user intervention. Although only one imaging region is used in FIGS. 6A-6E, in other examples, two or more imaging regions can be aligned. Additionally, stage motion can be minimized.

[0043] Example 8. Section localization, registration, and high-resolution imaging Referring to FIG. 7, an exemplary method 700 typically includes performing a first registration 702, using correlation based on a user-specified slice image to locate a slice of the sample. This may be conveniently referred to as a rough registration. At 704, the located slice is registered using a preview image that includes a selected portion of the located slice. The ROI of the preview image is used as a search template to register another preview image, and the search template is updated using the most recently registered preview image. This may be conveniently referred to as a fine registration. At 706, a final image (high resolution image) suitable for 3D tomographic reconstruction is obtained, forming a registered or registerable stack of images of at least a portion of the ROI.

[0044] Example 9. Preliminary alignment As discussed above, image registration is used to obtain a stack of registered images. Image registration is used during the preliminary stage, i.e., before recording the high-resolution images. Thus, the acquired image stack is already somewhat well registered, especially around the ROI where the refinement was performed. Thereby, the remaining position error is relatively small, e.g., <10 μm. After recording, another stack registration is required, but the deviation is <10 μm. In contrast, if high-resolution images are acquired with low-precision imaging field placement and stack registration is performed, the positioning error is typically >100 μm. In this approach, it would be necessary to add a border of >100 μm to the size of the recorded images to ensure that the ROI is captured in all sections. This can lead to a dramatic increase in imaging time. As mentioned above, in the disclosed approach, the acquired image stack is well registered around the ROI, and such a border is not necessary.

[0045] For example, for an ROI that is 40 μm x 40 μm square, with a desired resolution of 4 nm / pixel, the ideal image size is 10,000 x 10,000 pixels. For an imaged area positioning error of + / - 10 μm, the image size required to capture the ROI in all sections could be the ROI size + 2 x 10 μm, or 60 μm x 60 μm. This results in a recorded image size of 15,000 x 15,000 pixels, with an increase in imaging time of 15,000. 2 / 10,000 2 = 2.25. For an imaging area positioning error of + / - 100 μm, the image size required to capture the ROI in all sections can be ROI size + 2 × 100 μm = 240 μm × 240 μm. This results in a recording image size of 60,000 × 60,000 pixels, and an increase in imaging time of 60,000 2 / 10,000 2 = 36. It is therefore clear that requiring a large border can significantly increase image acquisition time.

[0046] Example 10. Section alignment and distortion 8A-8F show the processing of image sections placed on tape with and without image alignment, and illustrate the effect of pincushion distortion in the field of view. Other image artifacts such as image rotation, variable magnification, focus errors, and other image aberrations can be corrected in the same way, and distortion is illustrated as a convenient illustration. As discussed above, such image artifacts can lead to misalignment in image stacks based on sections of a specimen, and when stitching together images of one section to form a complete image of the section. FIGS. 8A and 8B show representative images 812, 832 obtained with nominally square fields of view corresponding to imaging fields 800, 820. (Imaging fields of view refer to the actual instrument field of view as imaged by the instrument). Imaging field 800 is the intended field of view and is free of imaging defects, while imaging field 820 exhibits pincushion distortion. In these examples, the section dimensions are approximately the same as the corresponding field dimensions. In FIG. 8A, a specimen section 808 is imaged using imaging fields of view 800A, 800B with overlap area 810. The image portions corresponding to the overlap area 810 are aligned, and the images obtained in the fields of view 800A, 800B can be precisely stitched together to produce image 812. Such imaging fields therefore allow images corresponding to sections at different locations within the field of view to be stitched together. In FIG. 8B, a specimen section 828 is imaged using distorted imaging fields of view 820A, 820B with overlap area 830. In the overlap area, the imaging fields are distorted, and the corresponding portions of imaging fields 820A, 802B have different distortions. The image portions corresponding to the overlap area 830 can be stitched together, but without precise alignment. The composite image corresponding to the imaging fields 820A, 820B produces a slice image 832, but with an error region 834 where the alignment may be inaccurate or portions of one or both of the stitched images may be missing. Thus, with such distorted imaging fields, images corresponding to slices at various locations within the field cannot be easily stitched together. Images of slices that are separately located within multiple fields of view present this stitching challenge.

[0047] FIG. 8C shows a series of sections, such as representative sections 842A-842E, placed on tape 838. As shown in FIG. 8D, sections 842A-842E are imaged in distorted imaging fields of view 840A-840H with various respective overlap areas 843-849. In this example, each section is fully imaged in two fields of view. Section dimensions may be 1-2 mm high and 2-3 mm long, often barely fitting into the field of view. This section placement is typically achieved by placing the sections on tape 838 during the cutting process. Each section is nearly fully imaged by simply stitching together the section's image at two locations within the imaging fields of view 840A-840H. Adjacent images correspond to overlap areas 843-849. These overlap areas correspond to image defects that limit stitching accuracy and result in image portions within offset areas 853-858 as shown in FIG. 8E that impair stitching. For example, image 852A of slice 842A includes portions 850A, 850B that correspond to fields of view 840A, 840B, respectively, and portions within offset area 853 that correspond to overlap area 843. Referring to FIG. 8G, preview images and alignment can be used to center slices 842A-842E or to align them relative to the fields of view within a single field of view as shown in FIG. 8F. A distorted imaging field of view 840 may include distortions, but produces images 871-875 that lack overlap areas that correspond to stitching errors, as shown in FIG. 8G.

[0048] 9A-9D show the alignment of a representative stack 900 including 16 specimen sections, such as representative sections 912A-912D, that contact adjacent sections. Such section arrangements may be referred to as section ribbons and may result from cutting the specimen without tape. Such sections may have, for example, an aspect ratio of 1:3, 1:4, or 1:5, and the sections may be longer and not as tall as sections created with tape, such as shown in FIG. 8C. As shown in FIG. 9A, the sections are imaged in imaging fields 902A and 902B, where each field images (at least partially) four different sections. Distortions in imaging fields 902A-902D correspond to offset regions 903, 905, 907 that impair stitching together image portions from the various imaging fields. Using the preview image, an aligned image stack 920 is created in which all slice images have approximately the same position in the field of view and do not require image stitching as shown in Figures 9C and 9D. For example, representative images 952A-952D of slices 912A-912D have a common alignment in a single field of view and do not require stitching. In this example, the slice edges are touching, so image 952B of slice 912B also includes image portions 961, 962 corresponding to slices 912A, 912C. Other slice images may similarly include portions corresponding to adjacent slices. When aligning a particular slice, a portion of the intended slice should be used, not a portion of an adjacent slice.

[0049] Considering the many possible embodiments to which the principles of the disclosed technology may be applied, it should be understood that the illustrated embodiments are merely preferred examples. We claim as our invention all that comes within the scope and spirit of the following claims.

Claims

1. 1. A method for producing an image in a scanning electron microscope, comprising: The processor: obtaining a set of preview images corresponding to a series of specimen sections arranged side by side; and processing to align each of the preview images of the set of preview images so as to register the preview images; The method comprises: obtaining an image including image areas corresponding to a plurality of specimen sections; processing the image to identify an area of ​​the image corresponding to the specimen section; establishing a location of the specimen section; each of the preview images corresponds to a respective specimen section; The method comprises: determining stage coordinates corresponding to alignment of the image area corresponding to the specimen section; The method further includes obtaining a region of interest (ROI) for each of the selected slices and generating a three-dimensional reconstruction of at least a portion of the ROI based on the registration of the preview image.

2. The method of claim 1 , wherein the preview images are processed for registration based on at least one feature in one or more of the preview images.

3. The method of claim 1 , further comprising: processing each preview image for registration based on a reference preview image selected from the set of preview images.

4. 4. The method of claim 1, further comprising obtaining a set of section images corresponding to a region of interest (ROI) of a specimen, the section images having a higher resolution than a resolution corresponding to the preview image, and the section images being aligned based on the registered preview image.

5. The method of any of claims 1 to 4, wherein each preview image is processed for registration based on correlation of a search template selected from the set of preview images.

6. 6. The method of claim 1, wherein the set of preview images includes N preview images 0,...,N, where N is an integer, and at least one preview image is processed for registration based on a search template selected from the set of preview images.

7. 7. The method of claim 6, wherein the i-th preview image is registered by comparison with the (i-1)-th preview image, where i is an integer greater than 1 and less than N.

8. 2. The method of claim 1, wherein processing each of the preview images of the set of preview images to register the preview images includes aligning the preview images or storing an image transformation corresponding to the alignment.

9. 2. The method of claim 1, wherein the preview images correspond to a first resolution, and further comprising obtaining a set of region of interest (ROI) images having a second resolution based on the registered preview images, the second resolution being greater than the first resolution.

10. A system for carrying out the method according to any one of claims 1 to 9, comprising: an optical imaging element positioned to obtain a synoptic image of a substrate including a plurality of specimen sections; a first image processor coupled to receive the overview image of the substrate and to locate image portions corresponding to the plurality of specimen sections; a charged particle beam imaging system (CPB imaging system) configured to generate a preview image corresponding to each selected portion of the specimen section; a second image processor coupled to receive the preview image and to determine an alignment of the preview image.

11. The system of claim 10 , wherein the CPB imaging system is configured to create an ROI image corresponding to each of the specimen sections and to align the ROI images based on the alignment of the preview images.

12. The system of claim 11 , wherein the first image processor and the second image processor are the same image processor.

13. 13. The system of claim 12, wherein the image of the substrate produced by the optical imaging element has a first resolution, the preview image has a second resolution greater than the first resolution, and the ROI image has a third resolution greater than the second resolution.

14. The system of claim 12 , wherein the first image processor is coupled to locate image portions corresponding to the plurality of specimen sections based on a correlation with a section template.

15. The system of claim 12 , wherein the second image processor is coupled to align the preview image based on correlation with one or more search templates or based on feature identification.

16. The system of claim 14 , further comprising a substrate stage, wherein the first image processor is coupled to determine a substrate stage location corresponding to alignment of the image of the specimen section, and the second image processor is coupled to determine a substrate stage location corresponding to alignment of the preview image.

Images