Method for recording nanometer resolution 3D image data from serial ultrathin sections of life science samples with an electron microscope

Abstract

Methods for recording nanometer resolution 3D image data from serial ultrathin sections of a life science sample using an electron microscope. Methods of aligning sample images of sample sections located on a substrate include obtaining optical or SEM images of the substrate, and positioning and aligning optical or SEM images of each sample section. The sample sections are then imaged using an SEM to obtain preview images, and a region of interest (ROI) in at least one of the preview images is selected. The preview images are processed such that at least the portions of the preview images proximate the ROI are aligned. Based on the alignment of the preview images, final SEM images of selected sample sections are obtained such that a set of images aligned in three dimensions are available. Image alignment can use cross-correlation with fixed or variable references that can be updated as sample section images are processed.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Application No. 62 / 868,617, filed June 28, 2019, which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to obtaining precisely matched subregions from multiple slices using scanning electron microscopy. Background Technology

[0003] Ultrastructural information about tissue samples has become increasingly important for life science research. While scanning electron microscopy (SEM) can produce high-resolution images, tissue samples do not efficiently generate the secondary or backscattered electrons required for SEM analysis. Therefore, samples are stained with heavy metals (e.g., osmium, lead) and subsequently trimmed or sectioned by embedding them in resin bottles. These resulting blocks are approximately 1 mm³ in size. Image formation in SEM is limited to the block surface because electrons cannot efficiently penetrate depths greater than approximately 30 nm into the block. To generate 3D image data, the block must be cut into continuous slices or thin layers removed from the block surface during imaging. Layer-removal-based methods, such as continuous block imaging or FIB-SEM dual-beam imaging, can damage the sample, and the user must acquire a large amount of data during imaging because the slices cannot be re-imaged. In cases where the sample is cut into slices used in so-called array tomography, each slice can be re-imaged as needed, and the user does not have to worry about sample damage. Commercially available ultrathin slicers can reproducibly cut 50 nm slices, and then place 100-300 consecutive slices on a conductive support, typically a 10 cm wafer or a metal plate with a maximum area of ​​10² cm².

[0004] High-resolution image data from entire tissue blocks is rarely required. It is also impractical: a 1 mm³ block recorded at a 10x10x50 nm pixel resolution corresponds to 3 PB of data, requiring 10⁵ days to record at a 3 µs beam dwell time. Target volumes or values ​​typically correspond to the size of one or several biological cells, ranging from 30³ µm³ to 100³ µm³. A major bottleneck in SEM imaging of sequential slices lies in navigating to the same 30² µm² to 100² µm² region in each of hundreds of sequential slices dispersed on the surface of a sample support. While SEM technicians can navigate in this way, it can be extremely time-consuming (from hours to days) to identify the appropriate slice locations and align these portions of the slice. For these and other reasons, alternative methods are needed. Summary of the Invention

[0005] Methods and apparatus are disclosed that allow for high-precision imaging of selected portions of specimen slices, i.e., minimizing inter-slice differences in the placement of the imaging region. In some instances, preview images are processed for registration based on at least one feature of one or more preview images. Alternatively, each preview image is processed for registration based on a search template selected from the set of preview images. In a typical instance, a set of specimen images associated with a Region of Interest (ROI) is obtained, wherein the resolution of the specimen images is higher than the resolution associated with the preview images. In some instances, each preview image is processed for registration based on its correlation with a search template image selected from the set of preview images. In a particular instance, the set of preview images comprises N preview images 0, ..., N, where N is an integer, and at least one preview image is processed for registration based on a search template image selected from the set of preview images. For example, the i-th preview image is registered by comparing it with the (i-1)-th preview image, where i is an integer greater than one and less than N. In some embodiments, each of the preview images in the set of preview images is registered by aligning the preview images or storing image transformations associated with alignment. In other instances, a preview image is associated with a first resolution, and a set of ROI images with a second resolution is obtained based on the registered preview image, wherein the second resolution is higher than the first resolution.

[0006] In other alternatives, an image containing image regions associated with multiple specimen slices is obtained, and the image is processed to identify the image regions associated with the specimen slices. Slice locations are established based on the identified image regions, where each of the preview images is associated with a corresponding specimen slice. In a representative example, platform coordinates associated with the alignment of the image regions associated with the specimen slices are obtained and stored. In a typical example, an ROI image is obtained for each of the selected slices, and a 3D reconstruction of at least a portion of the ROI is generated based on the alignment of the preview images or based on a higher resolution image aligned with the alignment of the preview images.

[0007] The system includes an imager positioned to obtain an overview image of a substrate comprising multiple sample slices. A first image processor is coupled to receive the overview image and locate image portions associated with the plurality of sample slices. A charged particle beam (CPB) imaging system is configured to generate a preview image associated with selected portions of each of the sample slices. A second image processor is coupled to receive the preview image and determine the alignment of the preview image. In some instances, the CPB imaging system is configured to generate a Region of Interest (ROI) image associated with each of the sample slices and to align the ROI image based on the alignment of the preview image. In a representative instance, the first and second image processors are the same image processor. In other instances, the overview image has a first resolution, the preview image has a second resolution higher than the first resolution, and the ROI image has a third resolution higher than the second resolution. According to some instances, the first image processor is coupled to locate image portions associated with the plurality of sample slices based on correlation with a slice template. In other instances, the second image processor is coupled to align the preview image based on correlation with one or more search templates or based on feature recognition. In some instances, a first image processor is coupled to determine the substrate platform position corresponding to the alignment of an image of a sample slice, and a second image processor is coupled to determine the substrate platform position corresponding to the alignment of a preview image.

[0008] The method includes using a processor to identify multiple specimen slices of a 3D sample from an overview image of the multiple slices of the 3D sample based on a slice template, wherein the overview image is an optical image associated with a first image resolution. The images of the identified multiple specimen slices are registered, and an optimized region (ROI) containing at least one set of selected images of the identified slices is selected. A preview image containing each of the optimized regions is obtained, the preview image being an electron-beam-based image with a second image resolution higher than the first image resolution. Feature recognition is used to register the preview images relative to each other. For each of the preview images, an electron-beam-based image associated with the ROI is obtained, wherein the electron-beam-based image has a third image resolution higher than the second image resolution. The registered electron-beam-based image associated with the ROI for each of the registered preview images is stored.

[0009] These and other features of the disclosed technology are set forth below with reference to the accompanying drawings. Attached Figure Description

[0010] Figure 1A-1B A representative method for aligning and registering images from slice samples is shown.

[0011] Figure 2A-2I This shows an image associated with the alignment of the sliced ​​image.

[0012] Figure 3 A representative imaging device for generating the registered image is shown.

[0013] Figure 4 This illustrates a representative computing environment for controlling image acquisition and processing.

[0014] Figure 5 This demonstrates a method for aligning images.

[0015] Figures 6A-6E A representative method for aligning sliced ​​images is shown.

[0016] Figure 7 This illustrates a representative method for obtaining a series of images for 3D tomography.

[0017] Figures 8A-8B The image shows slice imaging with and without distortion.

[0018] Figure 8C Showing a slice located on the strip.

[0019] Figure 8D-8E The image shows slice image artifacts associated with slice images obtained at different slice locations in the imaging field of view.

[0020] Figure 8F-8G This demonstrates how aligning slices within the imaging field of view can reduce or eliminate slice image artifacts associated with slice image stitching.

[0021] Figures 9A-9D The image shows the alignment of a strip of sample slice with a preview image in an image stack. Detailed Implementation

[0022] Definitions and Terms As used herein, “image” refers to an inspectable image presented on a display or otherwise available for user viewing, and a stored representation suitable for generating such an inspectable image. Examples of such representations include files in .jpg, .tiff, .bmp, and other formats stored on computer-readable media such as hard drives, memory, or otherwise. Images may be stored as intensity or other values ​​as a function of coordinates such as intensity I(x,y), where x,y are Cartesian coordinates. Other representations are possible, such as three-dimensional representations using Cartesian, polar, or other coordinate systems. For ease of description, methods are described as a sequence of specific steps, but in some cases these steps may be performed in a different order, and one or more steps may be performed simultaneously. In some cases, an image or a portion of an image is referred to as aligned or in alignment. As used herein, these terms refer to images of specimen slices processed by rotation and / or translation to overlap to correspond to the position and orientation in the specimen before slicing. Alternatively, these terms refer to images processed to identify rotations, translations, or other processing that permit the generation of images with the specimen position and orientation before slicing. For example, image coordinates can be updated such that all images are specified in a common coordinate system, or each image can be defined relative to its own or other coordinate systems, but with offsets and / or rotations that can be used to overlap or otherwise align the images as needed. In either case, the images can then be used to determine the specimen structure via a stack of slice images. Alignment can be used to determine the platform coordinates in an optical or CPB microscope to obtain suitable images.

[0023] In some instances, correlation with one or more reference images or templates is used to determine image alignment. Fixed or variable references or templates can be used. Typically, precise alignment of layer images uses reference images that can vary among slice images in a stack. For example, an image processed to be aligned relative to a reference can be used as a reference for aligning subsequent images. The reference image can change relative to at least one slice image in the stack, or at every other slice, every two, every three, every four, or other intervals. Features can be tracked between layers, and correlations between layers can be calculated, with the maximum correlation value used to indicate alignment.

[0024] Overview of the example This describes an example of processing slices of a specimen obtained using a microtome. Multiple slices are placed on a substrate, and a first alignment procedure (“coarse alignment”) is used to position the slices relative to each other based on comparison or correlation with a slice template, typically selected by the user from slice images contained in an overview image of the substrate. In a second alignment procedure (“fine alignment”), so-called preview images of some or all of the slices are obtained. A Region of Interest (ROI) template is selected from the selected slices, and the first preview image is aligned based on comparison or correlation with the ROI template. The ROI template is updated based on the aligned first preview image, and a second preview image is aligned based on comparison or correlation with the updated ROI template. This process is repeated for all preview images of interest. The ROI is then selected by the user, and high-resolution images of the alignment from some or all of the slices are obtained. In the examples discussed below, the overview image is an optical image, and the subsequent images (preview image and high-resolution image) are electron beam images. However, optical or charged particle beam images can be used, for example, images produced by electron microscopy, optical microscopy, optical scanning microscopy, ion beam imaging, or others.

[0025] Example 1. Tomographic imaging from a specimen slice The disclosed methods and apparatus can be used in 3D and other imaging of specimens, such as biological specimens. A representative method 100 is shown in Figure 1. At 102, a specimen block or other 3D specimen is obtained. The specimen block is then sliced ​​at 104 using, for example, a high-precision slicer, and the slices are typically arranged on a substrate in the order they were removed from the specimen block. A silicon wafer is a convenient substrate. A group of one or more substrates holding multiple specimen slices is referred to herein as an “array tomography sample” or simply a “sample.” The slices can be arranged arbitrarily, and the slices can be appropriately tracked to maintain the slice order as needed, but sequential ordering is generally more convenient. For example, the slice order can be stored at 105. From 1 mm 3 The slices obtained from the sample block are typically not fitted to a single substrate and may require 10–100 wafers. In most instances, the slices are arranged in rows extending along a parallel axis (e.g., from left to right), with each subsequent row beginning in a new row near the position of the initial slice in the previous row. Alternatively, a new row can begin by placing a slice near the last slice of the previous row; returning to the leftmost position after completing a row. Although not discussed in detail, sample blocks and slices can be stained or labeled as needed for optical or electron beam microscopy. For example, fluorescent immunolabeling and heavy metals can be used for optical and electron beam microscopy, respectively. In some cases, slices are arranged on strips, which are then positioned on a substrate. The substrate is coupled to a platform for positioning for optical and electron beam imaging, and feature and image positions can be specified based on platform coordinates and rotation.

[0026] Slices arranged on a substrate (i.e., the sample) are imaged at 108 using an optical imaging device such as a camera to obtain one or more overview images. For convenience, it is assumed below that only a single substrate and a single overview image are required. The overview image is processed at 110 to identify the slices and obtain their associated positions, typically as xy coordinates in a coordinate system with x and y axes in the plane holding the substrate surface of the slice. Each slice is assigned and the platform position (and orientation) is recorded at 111. Slices can be detected using a user-defined template and the overview image, for example, by cross-correlating with an optical image of the template. The template is typically selected from the slice images in the overview image, and this template is referred to herein as a “slice template.” Multiple portions of one or more overview images can be processed in parallel using cross-correlation to enable slice identification and faster slice placement. An image resolution of approximately 2 µm / pixel is used so that the image or portions thereof can be used in the correlation operation—in high-resolution images, the differences between sequential images may be too large for successful cross-correlation. Images of specific slices can be identified and used as slice templates for the positioning of all slices.

[0027] With the slices positioned and sorted, a slice preview image (typically using an electron beam) is obtained at 112. The preview image is associated with a portion of the slice containing the region of interest (ROI). In some instances, the preview image is obtained based on a user-provided contour drawing on the slice image, and a graphical user interface for this selection is provided. The preview image typically covers the entire slice and has a size defined by a slice template plus 1%, 5%, or 10%, but other sizes may be used. These preview images are typically obtained at a resolution higher than that used in slice positioning (e.g., better than 1µm / pixel). Preview images may exhibit variable artifacts such as distortion within the image, variable magnification between images, etc. Therefore, typically, no platform location provides a perfect or even satisfactory stack alignment of slices of the ROI at different locations in the preview image. However, a search template can be used to obtain a location suitable for each ROI using feature-based image alignment or correlation, which can be updated during processing such that image portions in and near each ROI image portion are aligned. The search template is typically selected as at least a portion of the first preview image. At 114, the preview image is aligned by tracking image features from one preview image to the next using the search template. In some instances, the features used for tracking are updated to accommodate variations in the sample after aligning one or more preview images, and the search template is updated after processing each preview image. In some instances, a selected preview image is used as a search template for cross-correlation with one or more other preview images. The search template can be updated as needed during processing. Due to image distortion and other image artifacts, each ROI typically requires independent feature-based or other alignments, but ROIs that are sufficiently close together may not require this. Acceptable proximity can be a function of the amount of image artifacts and proximity, and it may be more convenient to align each ROI using a dedicated feature-based alignment for each one. At 116, the alignment and registration values ​​of the preview images are obtained, and at 115, they are typically stored in a computer-readable medium.

[0028] Once the image stack is aligned, a high-resolution image (e.g., 2 nm / pixel) can be acquired at 118, and this high-resolution image is used in 3D reconstruction at 120. In some instances, preview image alignment can typically be repeated by acquiring and processing additional preview images at a higher resolution than the initial preview image. If additional ROIs are to be explored, processing returns to 112, and a suitable preview image associated with the additional ROI is acquired and processed. In this instance, the slices have already been located, and the relevant method steps are not required.

[0029] Example 2. Preview Image Alignment Figure 1BThis illustrates a representative method 150 for image stack alignment using a preview image or other image portions. For ease of explanation, refer to slices 0, 1, 2, ... N Image stack discussion Figure 1B ,in N This is an integer, with slice 0 being the topmost slice. At 152, a set of preview images is received, and at 154, a search template is selected, typically the preview image of slice 0 (or another slice). At 156, the preview image of slice i is selected, and at 158, it is typically compared and aligned with the search template using cross-correlation. At 160, the registration coordinates are typically stored as platform coordinates for subsequent alignment. At 162, it is determined whether additional preview images will be aligned, and if so, an updated search template is selected at 154. In some cases, the updated search template is the i-th image previously used for alignment, while in other instances, the initial reference image (image 0) is used. In other instances, different updated search templates are selected after processing 2, 5, 10, 20, 50, or 100 images, or the most recently used preview image can be selected. Changing the updated search template across the slice stack allows registration to be maintained even in cases where there are image features that change across the stack. In some cases, only the initially selected updated search template is sufficient, and any of the preview images can be used. More typically, continuously updating the search template from slice to slice allows for registration across hundreds of slices, even in the case of progressively altered biological structures. Upon alignment completion, the ROI portion of the preview image can be used to obtain the final high-resolution image and establish a 3D image of the volume of interest region.

[0030] Stack alignment can be performed from within the stack and does not necessarily have to start from the top or bottom portion. For example, the k-th preview image can be selected as the updated search template, and the (k-1)-th and (k+1)-th preview images can be aligned, and preferably the search template is refreshed. Preview images can be processed serially, or preferably multiple preview images can be processed in parallel.

[0031] The portion of the sample of interest typically extends only through selected sections of the specimen block. In these cases, images associated with all sections and all regions of the sections are not required. Users can conveniently select any section and region of interest using a graphical user interface.

[0032] Example 3. Representative Sample Treatment Figure 2A-2I The sample processing and ROI image alignment are shown. Figure 2AImage 200 shows a substrate 202 supporting a series of strips, such as representative strip 204, which hold sample slices, such as representative sample slice 206. Twelve strips are shown, but more or fewer strips may be used, and the sample slices may be located directly on the substrate 202. Multiple such substrates may be needed to hold all slices of the sample. Image 200 is an overview image, and a specific image portion 208 of image 210 containing a single slice is selected from the overview image 200. Image portion 208 is selected to serve as a slice template when identifying and locating other slices (typically using a slice template correlated with the overview image 200); then, the relative displacement associated with a large value of the correlation coefficient is identified and coordinates are obtained to establish the slice image position.

[0033] Figure 2B Image portion 212 is shown, which contains images of multiple slices that have been identified as indicated by frames such as frame 209, etc. Figure 2C The image further shows slices 220 and 221. The image of each of these slices is shown with the associated coordinate axes of a two-dimensional xy-coordinate system. For example, as... Figure 2D-2E As shown, slices 220 and 221 are associated with corresponding coordinate axes 230 and 231. In this example, the slice size is approximately 1 mm by 0.5 mm, and the positions of imaging areas 240 and 241 relative to the corresponding coordinate axes 230 and 231, as well as the rotation angles, are shown. r Typically, only the area within the slice that needs imaging is selected, and it can be done as follows: Figure 2F-2G The trace 250 shown in the image extends through all slices of interest, facilitating slice selection. Slices can be selected in various ways, such as drawing traces 250 on the displayed image of the substrate and slices using a computer-based pointing device. A cursor 256 can be used when creating traces 250, and the cursor 256 can be manipulated via computer-executable instructions for a mouse, trackpad, keyboard, or other device.

[0034] Selected slices are identified, and these slices have specified positions relative to each other, but are often not well aligned within their regions of interest (ROIs). Alignment and registration are limited by magnification errors, rotation errors, nonlinear distortion in the tiled SEM overview image already acquired with a large field of view, mismatches at slice boundaries, and unsatisfactory alignment and arrangement of slices in a stack. Figure 2H-2IAs shown, a portion 260 of slice 220 is selected, and a preview image 262 of region 261 is obtained. Preview image 262 and region 261 are selected as ROIs containing features of the specimen of interest. Preview image 262 is typically a higher resolution image than any previous image, such as an overview image. Preview images containing ROIs of other selected slices may also be obtained, and then the preview images are aligned using correlation or other processes as discussed above. In preview image-based alignment, an initial or previously aligned preview image or a portion thereof can be used as a search template, and the search template is updated after each preview image is processed. While preview images from each slice can be aligned, typically only the preview images indicated by trace 250 are aligned, but in some instances, hundreds of such preview images are selected.

[0035] After alignment with the preview image, a preview image can be provided to generate a 3D image of the ROI. Alternatively, this alignment allows for the acquisition of a higher resolution image, and the substrate platform can be used to properly position the slices. These additional images can also be aligned as needed. In any case, the resulting image stack allows for 3D reconstruction with minimal operator intervention.

[0036] In some instances, the user specifies the region surrounding the ROI and adjusts both the linear (e.g., xy coordinates) and rotation angles to match the search template. For example, an ROI is selected from a first slice as the search template, and the corresponding portion of a second slice is aligned by applying appropriate translation and rotation for matching. With the first and second slices aligned, the portion of the image surrounding the ROI in the second slice is selected for processing the third slice. This process continues until all slices of interest have been processed. In some instances, instead of adjusting the image, appropriate xy offsets and rotation angles are stored for subsequent image processing. As discussed above, other regions of the slice will require different offsets and rotations, and images of multiple regions can be acquired, processed, aligned, and stored.

[0037] Example 4. Imaging System refer to Figure 3 The imaging system 300 includes a system controller 302 coupled to ion beam sources 304 and electron beam sources 306, which generate ion beam 305 and electron beam 307, respectively. Corresponding scanners 312 and 314 are positioned to guide scanning ion beam 313 and scanning electron beam 315 relative to the sample 320, respectively. In some applications, images are obtained based on scanning electron beam 315, and scanning ion beam 313 is used for sample modification. However, images can be obtained by scanning either or both of ion beam 313 and scanning electron beam 315. In some cases, the imaging system includes only one of the electron beam source and ion beam source. For many biological samples, only the electron beam is required.

[0038] Sample 320 is fixed to platform 322, which is coupled to platform controller 324, which in turn is coupled to system controller 302. Platform 322 typically provides one or more translations, rotations, or tilts as guided by system controller 302. In response to a beam 326 of scanning ion beam 313 or scanning electron beam 315, an electron or ion detector 328 is directed to system electronics 330, which may include one or more analog-to-digital converters (ADCs), digital-to-analog converters (DACs), amplifiers, and buffers for controlling detector 328 and processing (amplifying, digitizing, buffering) the signals associated with detector 328. In other instances, a photon detector is used, which generates electrical signals that are further processed by the system electronics. In most practical instances, at least one ADC is used to generate a digitized detector signal, which can be stored as an image in one or more tangible computer-readable media (shown as image storage device 332). In other instances, image storage is remote via a communication connection such as a wired or wireless network connection. The beam 326 can be a scanning ion beam 313, a scanning electron beam 315, a scattering portion of secondary electrons, ions, or neutral atoms. An optical imager 351, such as a camera, is coupled to generate an image of the sample 320, for example, to generate a substrate image showing multiple substrate slices. As described above, such images can be processed to identify, locate, and align each slice using a charged particle beam (CPB) for further (typically higher resolution) imaging.

[0039] System controller 302 is coupled to memory 335, which stores processor-executable instructions for image processing, such as slice identification 336, correlation and feature alignment 340, selection of ROI and preview image region 338, storage and acquisition of overview image 339, and search template selection and update 341. It also provides a GUI 342 for including various functions to select which slices to process and defining the visible traces of the slices of interest. Images (both CPB and optical) can be stored in memory 332. Platform coordinates (including rotation) can also be stored in memory 332. System controller 302 establishes image acquisition parameters and communicates with platform controller 324. Sample images (e.g., preview images, slice images, substrate images, overview images) can be displayed on display 352, and system control and imaging parameters can be specified using values ​​internally stored in memory 335 or provided by the user using one or more user input devices 350.

[0040] It should be understood that Figure 3The layout is for illustrative purposes and the actual alignment of the various beam sources, optical camera 352, and CPB detector is not shown. Although a dual-beam (ion / electron) system is shown, one or both can be used, and in many practical examples such as electron microscopy, only the electron beam is used for imaging.

[0041] Example 5. Representative Computing Environment Figure 4 The following discussion aims to provide a brief general description of an exemplary computing environment in which the techniques of this disclosure can be implemented. Specifically, some or all portions of this computing environment can be used with the methods and apparatus described above to, for example, control beam scanning and image processing to identify and align sliced ​​images, preview images, and store images. Although not required, the disclosed techniques are described in the general context of computer-executable instructions executed by a personal computer (PC), such as program modules. Generally, program modules contain routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, the techniques of this disclosure can be implemented using other computer system configurations including: handheld devices, tablets, multiprocessor systems, microprocessor-based or programmable consumer electronics devices, network PCs, microcomputers, mainframes, etc. The disclosed techniques can also be practiced in distributed computing environments where tasks are performed on remote processing devices linked via a communication network. In a distributed computing environment, program modules may reside in both local and remote memory storage devices. In some cases, such processing is provided in a SEM (Search Engine Image Processor). The disclosed system can be used to control image acquisition and provides a user interface as well as act as an image processor.

[0042] See Figure 4 An exemplary system for implementing the disclosed technology includes a general-purpose computing device in the form of an exemplary conventional PC 400, the general-purpose computing device including one or more processing units 402, system memory 404, and a system bus 406 connecting various system components including the system memory 404 to the one or more processing units 402. The system bus 406 may be any of several types of bus structures using any of a variety of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus. The exemplary system memory 404 includes read-only memory (ROM) 408 and random access memory (RAM) 410. A basic input / output system (BIOS) 412 is stored in the ROM 408, the BIOS containing basic routines that facilitate the transfer of information between components within the PC 400.

[0043] The exemplary PC 400 further includes one or more storage devices 430, such as a hard disk drive for reading from and writing to a hard disk, a disk drive for reading from or writing to a removable disk, and an optical disc drive for reading from or writing to a removable optical disc (e.g., a CD-ROM or other optical media). Such storage devices may be connected to the system bus 406 via a hard disk drive interface, a disk drive interface, and an optical disc drive interface, respectively. The drives and their associated computer-readable media provide the PC 400 with non-volatile storage of computer-readable instructions, data structures, program modules, and other data. Other types of computer-readable media capable of storing data accessible by the PC, such as magnetic tape cassettes, flash memory cards, digital video discs, CDs, DVDs, RAM, ROM, etc., may also be used in the exemplary operating environment.

[0044] Several program modules may be stored in storage device 430, which includes an operating system, one or more applications, other program modules, and program data. Users can input commands and information into PC 400 via one or more input devices 440, such as a keyboard, and pointing devices, such as a mouse. For example, a user can input commands to initiate image acquisition or select whether, for example, optical flow or image difference will be used to locate the charging area. Other input devices may include digital cameras, microphones, joysticks, gamepads, disc satellite antennas, scanners, etc. These and other input devices are often connected to the one or more processing units 402 via a serial port interface connected to system bus 406, but may also be connected via other interfaces such as parallel ports, game ports, Universal Serial Bus (USB), or wired or wireless network connections. Monitor 446 or other types of display devices are also connected to system bus 406 via an interface such as a video adapter and can display, for example, one or more slice images (i.e., images used for identifying and locating slices), preview images, ROI images, or other raw or processed images, such as aligned images or images with displayed values ​​of translation and rotation required for alignment. Monitor 446 can also be used to select slices for processing or specific image alignment and alignment procedures, such as correlation, feature recognition, and preview area selection or other image selection. Other peripheral output devices may be included, such as speakers and printers (not shown).

[0045] PC 400 can operate in a networked environment using a logical connection to one or more remote computers (e.g., remote computer 460). In some instances, it includes one or more network or communication connections 450. Remote computer 460 can be another PC, server, router, network PC, or peer device or other common network node, and typically includes many or all of the elements described above with respect to PC 400, but... Figure 4 Only memory storage device 462 is shown. Personal computer 400 and / or remote computer 460 can be connected to a logical local area network (LAN) and a wide area network (WAN). Such networking environments are common in offices, enterprise-wide computer networks, intranets, and the Internet. In some instances, alignment image stacks are transmitted to remote systems for 3D image reconstruction or other processing.

[0046] like Figure 4 As shown, memory 490 (or portions of this or other memory) stores processor-executable instructions for image acquisition to establish dose, frame time, beam current, scan rate, and image processing. Furthermore, memory 490 includes processor-executable instructions for setting cross-correlation, image alignment such as image rotation and translation, selection of reference images and ROIs, and recording platform coordinates for alignment. In some instances, the processor-executable instructions produce the displayed image, which demonstrates slice recognition, preview image processing, and acquisition of additional images.

[0047] Example 6. Representative methods for aligning sliced ​​images Figure 5 A method 500 is illustrated for generating a set of aligned images of selected portions of multiple sample slices. At 502, a substrate image (typically an optical image) containing multiple slice images is acquired and displayed, and at 502, the positioned slice images are acquired (typically an SEM image) and displayed. At 506, a preview image is acquired, and at 508, an optimized region is selected from one or more slice images. At 510, the image position is optimized, and at 512, the aligned images are stored or output. Alternatively, appropriate translations and rotations can be stored for each image, and misaligned images can be output along with these translations and rotations. In some cases, method 500 continues with minimal user input beyond the selection of the ROI.

[0048] Example 7. Sample preparation and processing for computed tomography. Figures 6A-6E The example shown is 600 representative methods. See also... Figure 6AAnimal or tissue biopsies are performed at position 602, and samples are prepared at position 604 for imaging. The biopsy tissue is trimmed to a block smaller than 2 mm³. Furthermore, the block undergoes some or all of chemical fixation, heavy metal staining, and resin infiltration and curing, and is trimmed for serial slicing. A typical resin block has a 0.5–2 mm² front surface area and is 0.5–2 mm thick. At position 606, the resin block is serially sliced ​​into multiple slices 40–100 nm thick using an ultramicrotome. The slices are collected on a substrate such as a strip, glass plate, or wafer. If a strip is used, one or more strips of varying lengths are glued to a wafer or metal plate. This collection of slices on the substrate is called an “array tomography sample” or simply a “sample.” In some cases, the collection may extend to multiple substrates, all of which may be included in the array tomography sample. The sample is then prepared at position 610 for unattended data acquisition, as described below. Figures 6B-6D Further discussion was conducted, and a high-resolution image was obtained at 670.

[0049] Figure 6B The diagram illustrates, for example, the data acquisition preparation method 620 used at 610 above. At 622, the sample is placed on the microscope specimen platform, and at 624, the coordinates of all sections on the sample are determined. See below. Figure 6C The determination of slice coordinates is further discussed. After obtaining the slice coordinates, at position 626, the coordinates of the Region of Interest (ROI) in all or selected slices are obtained. At position 628, the imaging area of ​​the selected slice is created. In some cases, an optical image of the entire array tomographic sample is recorded in the SEM. This image shows all or most of the slices at a coarse resolution and can be used to define the area for obtaining an overview image.

[0050] See Figure 6C The method 630 for determining slice coordinates includes obtaining an optical image of the sample at 632. This image can be used to define the sample area targeted by the overview image obtained at 636. The overview image shows multiple slices and can be a single image or an image mosaic. The overview image has a relatively coarse resolution, i.e., 1-2 µm pixel size, allowing the entire substrate to be imaged within a reasonable time. A correlation between the number of images and the number of slices is not required. One image can show several slices, or multiple images may be needed to show a single slice. This depends on the maximum field of view of the microscope and the size of the slices. In some cases, overview images are obtained by means other than SEM. After importing such overview images, they must be aligned so that the slice positions in the imported image correspond to the plateau coordinates of the same slice on the array tomography sample. This can be achieved through alignment, where two or three landmarks visible in the two SEM images and the imported image are manually matched.

[0051] The overview image is used as follows. At 636, the user selects a region (usually a rectangle) from the overview image. The selected region is copied from the overview image to serve as a slice template. In the automated program, at 638, the slice template is correlated with the overview image to determine the slice position in the overview image. At 640, the matching position in the overview image is translated to a position in the platform coordinate system, i.e., the platform position for each slice is stored. The platform is moved to one of the stored positions so that the corresponding slice is centered below the microscope imaging system (“magnetic pole piece”). Any slices not discovered by the automated program can be added by the user by marking them in the overview image. Any incorrectly identified slices (i.e., non-slices) are marked as false positives and removed from the list of discovered slices.

[0052] Figure 6D A method 650 for determining the coordinates of a Region of Interest (ROI) across all slices is shown. At 652, a slice preview image is acquired. The slice preview image typically has the same or higher resolution as the overview image. In some instances, a pixel size of 200-800 nm is used. At 654, a slice preview image is selected, and at 656 it is determined whether the selected slice preview image is a slice preview image of the first slice. If so, the ROI is labeled in the first slice preview image by (for example) drawing an outline on a display device using a computer pointing device, and at 658 the ROI is stored as a search template. At 660, a match for the search template in the next slice preview image is identified, and at 662 the associated position and angle are stored and / or translated into platform coordinates. If additional slices are determined to be processed at 664, the next slice preview image is selected at 654, and at 666 the search template is updated to the matching position in the previously evaluated slice preview image. After processing each slice preview image to match the search template, the matching region for that slice preview image is set at the search template. In this way, the search for the matching region is optimized at each step.

[0053] Once all slices have been processed for alignment (as discussed above), then use Figure 6E The method 680 shown obtains an image. At 682, the user defines an imaging region on any one of the slices and selects corresponding regions for other slices. At 684, the image of the imaging region is obtained. In a typical example, the pixel resolution is between 5 and 50 nm, and the field of view is between 30²µm² and 100²µm². Because the platform coordinates of each slice have been obtained, method 680 can be executed by the processor without user intervention. Figures 6A-6E In some cases, a single imaging region is used, but in others, two or more imaging regions can be aligned. Furthermore, platform movement can be minimized.

[0054] Example 8. Slice localization, alignment, and high-resolution imaging See Figure 7 A typical method 700 includes performing a first alignment 702, which typically uses slices of relevant localization samples based on user-identified slice images. This may be referred to as coarse alignment for convenience. At 704, the localized slice is aligned using a preview image containing a selected portion of the localized slice. The ROI of the preview images is selected as a search template to align with other preview images, and the search template is updated using the most recently aligned preview image. This may be referred to as fine alignment for convenience. At 706, a final image (e.g., a high-resolution image) is obtained, which forms at least a portion of the ROI and is suitable for 3D tomographic reconstruction of an aligned or alignable image stack.

[0055] Example 9. Preparing for Alignment As discussed above, image alignment is used to obtain a stack of aligned images. Image alignment is used during the preparation phase, i.e., before recording high-resolution images. Therefore, the acquired image stack is already properly and well aligned, especially around the ROIs in which optimization is performed. Consequently, the residual positional error is relatively small, for example, <10 µm. Another round of stack alignment is required after recording, but the shift is <10 µm. In contrast, acquiring high-resolution images and then performing stack alignment with low-accuracy imaging area placement typically results in a positioning error >100 µm. Using this method, a boundary >100 µm needs to be added to the size of the recorded image to ensure ROI capture in all slices. This leads to a significant increase in imaging time. As mentioned above, in the disclosed method, the acquired image stack is well aligned around the ROI, and this boundary is not required.

[0056] For example, for a ROI of 40 µm x 40 µm squared and a desired resolution of 4 nm / pixel, the ideal image size is 10,000 x 10,000 pixels. With an imaging region localization error of + / - 10 µm, the image size required to capture the ROI across all slices will be ROI size + 2 x 10 µm, or 60 µm x 60 µm. Therefore, the recorded image size is 15,000 x 15,000 pixels; the increase in imaging time is 15,000² / 10,000² = 2.25. With an imaging region localization error of + / - 100 µm, the image size required to capture the ROI across all slices will be ROI size + 2 x 100 µm = 240 µm x 240 µm. Therefore, the recorded image size is 60,000 x 60,000 pixels; the increase in imaging time is 60,000² / 10,000² = 36. Therefore, it is clear that requiring large boundaries may significantly increase image acquisition time.

[0057] Example 10. Slice alignment and distortion Figures 8A-8F The processing of image slices located on a strip is illustrated with and without image alignment, and the pincushion distortion effect in the field of view is demonstrated. Other image artifacts, such as image rotation, variable magnification, focus errors, and other image aberrations, can be similarly compensated for, and the distortions are shown for illustrative purposes. As previously discussed, such image artifacts can lead to misalignment in the image stack of sample-based slices and in the process of stitching together images of individual slices to form a complete image of the slice. Figures 8A-8B Representative images 812 and 832, obtained with a nominal square field of view and associated with imaging fields of view 800 and 820, are shown. (Imaging field of view refers to the actual instrument field of view imaged by the instrument). Imaging field of view 800 is a given field of view with no imaging defects, while imaging field of view 820 exhibits pincushion distortion. In these examples, the slice size is approximately the same as the corresponding field of view size. Figure 8A In this process, the sample slice 808 is imaged using imaging fields of view 800A and 800B with overlapping regions 810. The image portions associated with the overlapping regions 810 are aligned, and the images obtained with fields of view 800A and 800B can be accurately stitched together to produce image 812. Therefore, in this case of imaging fields of view, images associated with slices at different locations within the field of view can be stitched together. Figure 8B In this process, a sample slice 828 is imaged using distorted imaging fields of view 820A and 820B with an overlapping region 830. In the overlapping region, the imaging fields of view are distorted, and this distortion is different in the corresponding portions of the imaging fields of view 820A and 820B. Image portions associated with the overlapping region 830 can be combined, but are not accurately aligned. The combined image associated with the imaging fields of view 820A and 820B produces a slice image 832, but with an error region 834, in which one or both portions of the incorrectly aligned or stitched images may be missing. Therefore, in the case of this distorted imaging field of view, images associated with slices at different locations within the field of view are not easily stitched together. This stitching difficulty exists for images of slices located in multiple fields of view in different ways.

[0058] Figure 8C Showing a series of sections, such as representative sections 842A-842E, located on strip 838. For example... Figure 8DAs shown, slices 842A-842E are imaged in distorted imaging fields of view 840A-840H with different corresponding overlapping regions 843-849. In this example, each slice is fully imaged in both fields of view. Slice sizes can be 1-2 mm high by 2-3 mm long, and often fit very little within the field of view. This slice arrangement is typically produced during the cutting process when the slice is placed on strip 838. Each slice is usually fully imaged only by stitching together the images of the slices at two locations within the imaging fields of view 840A-840H. Adjacent images are associated with overlapping regions 843-849. These overlapping regions are associated with image defects, thus limiting stitching accuracy and, as in... Figure 8E Image portions are generated in offset regions 853-858 that hinder stitching. For example, image 852A of slice 842A includes portions 850A and 850B associated with fields of view 840A and 840B, respectively, and a portion in offset region 853 associated with overlapping region 843. See also Figure 8G Using the preview image and alignment, slices of 842A-842E can be centered relative to the field of view or otherwise aligned and positioned as shown. Figure 8F Within a single field of view, the distorted images 871-875 generated by the imaging field of view 840 may contain distorted but lack overlapping areas associated with stitching errors, such as... Figure 8G As shown.

[0059] Figures 9A-9D This diagram illustrates the alignment of a representative stack 900 comprising 16 specimen slices, such as representative slices 912A-912D, that are in contact with adjacent slices. This slice arrangement can be referred to as a strip of slices and can be produced during specimen cutting without strips. Such slices may have an aspect ratio of, for example, 1:3, 1:4, or 1:5, and the slices may, for example... Figure 8C The slices produced by the strip shown are longer, but shorter than the slices produced by the strip described above. For example... Figure 9A As shown, the slices are imaged in imaging fields of view 902A-902B, where each field of view images four different slices (at least partially). Distortions in imaging fields of view 902A-902D are associated with offset regions 903, 905, and 907 that prevent the image portions from different imaging fields of view from being stitched together. Using the preview image, an aligned image stack 920 is generated, where all slice images have substantially the same position in the field of view, without requiring... Figure 9C-9DThe image stitching is shown. For example, representative images 952A-952D of slices 912A-912D have common alignment in a single field of view and do not require stitching. Because the slice edges touch in this example, image 952B of slice 912B also includes image portions 961 and 962 associated with slices 912A and 912C. Other slice images may similarly contain portions associated with neighboring slices. When performing alignment of a particular slice, portions of the given slice should be used, not portions of neighboring slices.

[0060] Given that the principles of the disclosed technology are applicable to many possible embodiments, it should be recognized that the illustrated embodiments are merely preferred examples. We claim that the entirety of the contents within the scope and spirit of the appended claims is our invention.

Claims

1. A method for acquiring matching regions of a sample slice to generate 3D image data, comprising: utilizing a processor: Obtain a set of preview images associated with a series of specimen slices into which the specimen block is cut; Each of the set of preview images is processed to register the preview images by using a search template to track image features from one preview image to the next; as well as Obtain a set of slice images associated with the region of interest (ROI) of the specimen; The process involves updating the image features used for tracking after aligning one or more preview images to adapt to changes in the sample slice, and updating the search template after processing each preview image. The sliced ​​image has a higher resolution than that associated with the preview image, and the sliced ​​image is aligned based on the registered preview image.

2. The method of claim 1, wherein the preview image is processed for registration based on at least one feature of one or more of the preview images.

3. The method of claim 1, further wherein each preview image is processed for registration based on a reference preview image selected from the set of preview images.

4. The method according to any one of claims 1-3, wherein each preview image is processed for registration based on the correlation of a search template selected from the set of preview images.

5. The method according to any one of claims 1-3, wherein the set of preview images comprises N preview images 0, ..., N, where N is an integer, and at least one preview image is processed for registration based on a search template selected from the set of preview images.

6. The method of claim 5, wherein the i-th preview image is registered by comparing it with the (i-1)-th preview image, where i is an integer greater than one and less than N.

7. The method of claim 1, wherein processing each of the preview images in the set of preview images to register the preview image includes aligning the preview image or storing image transformations associated with the alignment.

8. The method of claim 1, wherein the preview image is associated with a first resolution, and the method further comprises obtaining a set of regions of interest (ROI) images with a second resolution based on the registered preview image, wherein the second resolution is higher than the first resolution.

9. The method of claim 1, further comprising: Obtain an image containing image regions associated with multiple sample slices; The image is processed to identify the image region associated with the sample slice; as well as Establish the location of the sample slice, wherein each of the preview images is associated with the corresponding sample slice.

10. The method of claim 9, further comprising determining platform coordinates associated with the alignment of the image region associated with the sample slice.

11. The method of claim 1, further comprising obtaining a region of interest (ROI) image for each of the selected slices, and generating a three-dimensional reconstruction of at least a portion of the ROI based on the alignment of the preview image.

12. A system for acquiring matched sub-regions of a sample slice to generate 3D image data, comprising: An optical imager is positioned to obtain an overview image of a substrate containing a block of samples cut into multiple sample slices. A first image processor is coupled to receive the overview image of the substrate and locate the image portions associated with the plurality of sample slices; A charged particle beam (CPB) imaging system configured to produce preview images associated with selected portions of each of the specimen slices; as well as A second image processor is connected to receive the preview image and determine the alignment of the preview image by tracking image features from one preview image to the next using a search template; The process involves updating the image features used for tracking after aligning one or more preview images to adapt to changes in the sample slice, and updating the search template after processing each preview image. The charged particle beam (CPB) imaging system is configured to generate a region of interest (ROI) image associated with each of the specimen slices, and to align the ROI image based on the alignment of the preview image. The image of the substrate generated by the optical imager has a first resolution, the preview image has a second resolution higher than the first resolution, and the region of interest (ROI) image has a third resolution higher than the second resolution.

13. The system of claim 12, wherein the first image processor and the second image processor are the same image processor.

14. The system of claim 12, wherein the first image processor is coupled to locate image portions associated with the plurality of sample slices based on their correlation with the slice template.

15. The system of claim 12, wherein the second image processor is coupled to align the preview image based on correlation with one or more search templates or based on feature recognition.

16. The system of claim 14, further comprising a substrate platform, wherein the first image processor is coupled to determine a substrate platform position corresponding to the alignment of the image of the sample slice, and the second image processor is coupled to determine a substrate platform position corresponding to the alignment of the preview image.

17. A method for obtaining a matching sub-region of a sample slice, comprising: Using the processor, Identifying multiple slices of a 3D sample from an overview image of multiple slices of a 3D sample slice based on a slice template, wherein the overview image is an optical image associated with a first image resolution, and the slice template is based on an image portion associated with a selected slice; Register images of multiple identified sample slices; Select at least one region of interest for the specimen in a slice; Obtain a preview image containing the region of interest in a set of selected slices, wherein the preview image is an electron beam-based image with a second image resolution higher than the first image resolution; The preview images are registered relative to each other by tracking image features from one preview image to the next preview image using a search template; Obtain an electron beam-based image of each of the preview images associated with a region of interest (ROI), wherein the electron beam-based image has a third image resolution higher than the second image resolution; as well as Each of the registered preview images is a registered electron beam-based image associated with the region of interest (ROI); The process involves updating the image features used for tracking after aligning one or more preview images to accommodate variations in the sample slices, and updating the search template after processing each preview image.

18. A method for obtaining a matching sub-region of a sample slice, comprising: Obtain an overview image containing slice images associated with multiple slices of the sample, the overview image having a first resolution; Each slice in the slice image is located based on the slice template; Obtain a preview image of a set of selected slices associated with the region of interest, the preview image having a second resolution higher than the first resolution; The alignment of a preview image is determined based on at least one search template associated with the preview image by using a search template to track image features from one preview image to the next preview image, wherein the alignment of an individual preview image is based on the preview image for which the alignment is determined; as well as A set of final slice images is obtained, wherein the final slice images are aligned based on the determined alignment of the preview image, and wherein the final slice images have a third resolution higher than the second resolution; The process involves updating the image features used for tracking after aligning one or more preview images to accommodate variations in the sample slices, and updating the search template after processing each preview image.

Images