Automated image processing for electron microscopy

Abstract

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for determining the 3D structure of a type of particle. The system obtains one or more two-dimensional (2D) micrograph images of the sample, and performs a process for generating a set of representative 2D projections of the type of particle from the 2D micrograph images. The system obtains metadata for each class representation image, and selects, using the metadata, a set of representative 2D projections from the class representation images. The system reconstructs a 3D image of the type of particle using the set of representative 2D projections, and outputs the 3D image of the type of particle

Description

AUTOMATED IMAGE PROCESSING FOR ELECTRON MICROSCOPYCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to EP Application No. 24315049.7, filed on February 20, 2024. US Provisional Patent Application No. 63 / 663,224, filed on June 24, 2024, and EP Application No. 24315390.5, filed on August 27, 2024, the disclosures of all of which are hereby incorporated by reference in their entirety.TECHNICAL FIELD

[0002] The present invention in general relates to systems and methods for automated image processing for reconstructing three-dimensional (3D) particle images for electron microscopy (EM), and in particular, relates to identifying two-dimensional (2D) projections of particles in the EM images for 3D reconstruction.BACKGROUND

[0003] Cryo-electron microscopy (Cryo-EM) involves the rapid freezing of specimens to preserve their native conformations, thereby overcoming structural distortions induced by conventional fixation methods. Utilizing electron microscopy, Cryo-EM captures high- resolution images of these flash-frozen specimens, and subsequent computational analysis, including single-particle reconstruction, facilitate the generation of three-dimensional structures with near-atomic resolution.

[0004] In contrast to other structural biology techniques like X-ray crystallography, Cryo- EM eliminates the need for preparing crystals, making it particularly advantageous for studying flexible molecules and large protein complexes. Furthermore. Cryo-EM enables capturing multiple conformations of a molecule in a single experiment, providing valuable insights into its flexibility and function.SUMMARY

[0005] This disclosure describes methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for determining a three-dimensional (3D) structure of a type of particle in a sample.

[0006] In one aspect, this disclosure provides a method for determining the 3D structure of a type of particle. The method can be implemented by a system including one or more computers. The system obtains one or more two-dimensional (2D) micrograph images of the sample, and performs a process for generating a set of representative 2D projections of the type of particle from the 2D micrograph images. To generate the set of representative 2D projections, the system identifies a plurality of 2D particle projections in the 2D micrograph images, classifies the plurality of 2D particle projections into a set of classes, determines, for each class in the set of classes, a respective class representation image representing the 2D particle projections in the respective class; obtains respective metadata for each class representation image, the respective metadata including one or more of: an average pixel value, a maximum pixel value, a minimum pixel value, a pixel value standard deviation, or a pixel value signal-to-noise ratio (SNR) of pixels in the class representation image; and selects, using the respective metadata for each class representation image, the set of representative 2D projections from the class representation images of the set of classes. The system reconstructs a 3D image of the type of particle using the set of representative 2D projections, and outputs the 3D image of the type of particle.

[0007] In some implementations, to determine the class representation image representing the 2D particle projections in the respective class, the system performs an averaging across the 2D particle projections in the respective class to obtain an averaged image as the class representation image.

[0008] In some implementations, the process for generating the set of representative 2D projections is performed for a plurality of iterations. In each iteration after the first iteration, the set of 2D particle projections in the 2D micrograph image are identified using templates based on the set of representative 2D projections selected by the preceding iteration.

[0009] In some implementations, the respective metadata further includes a class distribution value characterizing a number of 2D particle projections in the class for the respective class representation image.

[0010] In some implementations, to select the set of representative 2D projections from the class representation images of the set of classes, for each respective class, the system determines one or more parameter values from the respective metadata of the respective class representation image, determines whether each of the one or more parameter values is within a respective range defined by one or more respective threshold values for the parameter value. In response to determining that each of the one or more parameter values is within the respective range, the system includes the respective class representation image in the selection. In response to determining that at least one of the parameter values is not within the respective range, the system excludes the respective class representation image from the selection. In some cases, for each respective parameter value, the respective range is determined based on a respective predefined quantile of the respective parameter value. In some cases, the one or more parameter values used for selecting the set of representative 2D projections include the average pixel value and the pixel value signal -to-noise ratio. In some cases, the pixel value signal-to-noise ratio is computed as a ratio of the maximum pixel value to the minimal pixel value.

[0011] In some implementations, to select the set of representative 2D projections from the class representation images of the set of classes, for each respective class representation image, the system generates a respective model input for the respective class representation image using the respective metadata, processes the respective model input using a machinelearning model to generate respective output, and determines whether to include the respective class representation image in the selection based on the output. For example, to generate the respective model input, the system can generate a respective vector using the average pixel value, the maximum pixel value, the minimum pixel value, the pixel value standard deviation, or the pixel value SNR, and a class distribution value, and generate the respective model input from the respective vector.

[0012] In some implementations, the 2D micrograph images include electron microscope (EM) images of the sample. In some cases, the 2D micrograph images include a set of EM images of the sample taken from different angles. In some cases, before performing the process for generating the set of representative 2D projections, the system processes the 2D micrograph images to correct motion artifacts. In some cases, before performing the process for generating the set of representative 2D projections, the system estimates a contrasttransfer function (CTF) of the 2D micrograph images and processes the 2D micrograph images to reduce blurring caused by the CTF.

[0013] In some implementations, the system further receives configuration data defining a workflow for performing operations of the process for generating the set of representative 2D projections and reconstructing the 3D image using the set of representative 2D projections, and performs the operations based on the workflow defined in the configuration data. In some cases, the configuration data defines one or more loops or repeated steps in the operations. In some cases, the configuration data defines one or more algorithms or options used in the operations. In some cases, the configuration data includes one or more tags that mark the repeated steps.

[0014] This disclosure also provides a system including one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the method described above.

[0015] This disclosure also provides one or more computer storage media storing instructions that when executed by one or more computers, cause the one or more computers to perform the method described above.

[0016] The subject matter described in this disclosure can be implemented in particular embodiments so as to realize one or more advantages.

[0017] Cryo-EM has emerged as a pivotal technique in diverse fields, including structural biology7, drug development, vaccine development, and materials science. Its ability to provide detailed structural information about proteins and viruses has significantly advanced our understanding of their roles in health and disease. This structural insight, derived from Cryo- EM, not only guides the precise targeting of new drugs, but also aids in vaccine development by elucidating virus structures. Moreover, the exploration of biomolecular structures inspires the design of new materials with unique properties, expanding the impact of Cryo-EM across diverse scientific disciplines.

[0018] There are a few challenges in the conventional techniques for processing EM micrographs to reconstruct 3D particle structures. The EM micrographs received from an image capture device contain artifacts caused by, for example, contaminants such as dust or other debris, as well as ice contamination which occurs in the vitrified ice layer that surroundsthe sample. The artifacts can also be caused by out-of-focus particles, over-exposed or underexposed areas, uneven illumination, charge build-up from electron charging, optical aberration, noises, and so on. In order to obtain a high-quality 3D reconstruction of the target particle, it is critical to obtain representative 2D projections of the target particle that capture the structural information of the target particle with as little artifact as possible. This process typically includes selecting representative 2D projections of the target particle from a set of candidate 2D projections. This Specification provides techniques for improving the selection process. Instead of relying solely on image features for selection, the provided system extracts metadata from each class representation image, such as average pixel value and signal -to-noise ratio (SNR), and selects representative projections based on the metadata, using either (i) exclusion criteria to filter out projections with outlier values, or (ii) a machinelearning model (e.g., a random forest) to predict which projections best represent the target particle.

[0019] Metadata-based projection selection offers advantages over relying solely on image features. This is because image features can be variable and sensitive to certain factors, including (i) image scale — changing the image size can significantly alter pixel-level features, (ii) image normalization — adjusting pixel intensity ranges can impact feature values, and (iii) positional changes - shifting particles within images affects feature extraction. Metadata parameters are more robust to these variations. They capture essential characteristics of the images such as average pixel value, signal-to-noise ratio, class distribution, etc. These characteristics are less sensitive to minor image manipulations or variations in particle position. Thus, metadata-based selection provides a more stable and reliable basis for selecting representative particle projections and reduces the risk of selecting projections based on features that might be artifacts of image processing or particle positioning, leading to a more accurate and robust 3D reconstruction. This more effective selection process can also reduce the number of iterations needed for obtaining high-quality particle reconstructions, thus reducing the need for computation resources and improving the speed of the computation process without needing manual input by an expert user.

[0020] In addition, in some implementations, the provided system uses user-defined workflow configuration data to automate the reconstruction workflow. By using the user- defined workflow configuration data, the users can define and arrange the sequence of steps, loops, algorithms, and options as needed to tailor the workflow to specific reconstructionrequirements. The workflow can incorporate repeatable steps without limitations, enabling iterative processes and optimization. The workflow configuration data can use tags to organize repeat steps to avoid conflicts. The pre-defined workflow can execute automatically, reducing manual intervention and maximizing computational resource utilization, and thus reducing the need for computation resources and improving the speed of the computation process. Users can initiate workflows at any time, eliminating the need for careful planning around job completions. The workflow configuration can be based on expert knowledge, enabling users without extensive experience to effectively reconstruct complex structures. Thus, the user-defined workflow configuration increases efficiency and productivity through automation, enhances flexibility to adapt workflows to diverse projects and requirements, optimizes computational resource usage, and expands the accessibility of Cryo-EM studies to a broader user base.

[0021] The details of one or more embodiments of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG. 1 shows an example environment for reconstructing 3D images for a Cryo-EM sample.

[0023] FIG. 2 is a flow diagram of an example process for reconstructing 3D images from micrographs.

[0024] FIG. 3 is a flow diagram of an example process for generating representative 2D projections.

[0025] FIG. 4 shows examples of representative 2D particle projections and their metadata.

[0026] FIG. 5 is a block diagram of an example computer system.

[0027] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0028] FIG. 1 shows an example environment 100 for an image processing system 130 to reconstruct 3D images 140 of a sample 110.

[0029] The sample 110 can be any appropriate specimen and can include, for example, purified protein molecules, membrane proteins, viral particles, and cellular components such as organelles or macromolecular complexes. Different types of specimens can be prepared using sample-specific processes. For example, purified protein molecules can undergo concentration and stabilization to optimize particle density. Membrane proteins can undergo detergent solubilization and incorporation into lipid nanodiscs or amphipols. Viral particles typically require purification from host cells, often employing gradients or affinity chromatography. Cellular components like organelles or macromolecular complexes may be processed using homogenization and fractionation techniques. The sample 110 undergoes vitrification, which includes rapid freezing into liquid ethane or propane to preserve native hydration and avoid ice crystal formation for high-resolution studies. The sample 1 10 can be placed on a cryo grid. The cryo grid is a support platform used to hold and position the sample 100.

[0030] The image capture system 120 includes an electron microscope (EM). During image capture, the EM scans the sample 110 placed on the cryo grid using a focused electron beam, inducing elastic scattering of electrons. The scattered electrons are focused within the electron microscope through a series of magnetic lenses and projected onto a high-resolution detector, generating a two-dimensional (2D) projection image of the sample 110. Multiple images are typically captured at varying defocus levels and from different viewing angles. These viewing angles are due to the rotational distribution of the particles within sample. In some cases, multiple images are captured with a fast frame rate, so post-processing can be used to mitigate beam-induced motion. In an illustrative example, the detector collects at 320 fps over each image. In post processing, each motion-corrected image is processed from a large number of sub-images.

[0031] In theory, the resolution of a Cryo-EM projection image is limited by the wavelength of the electron beam, which is much smaller than the typical size of biomolecules. For example, with a ty pical accelerating voltage of 200 kV, the electron wavelength is around 0.037 nm. In practice, several factors may reduce the resolution of a single projection image,such as beam-induced motion, defocus, detector noise, sample heterogeneity, and large particle counts.

[0032] The image processing system 130 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which an image processing method can be implemented. As will be described in more detail with references to FIG. 2 and FIG. 3, the image processing system 130 processes the 2D projection images (i.e., 2D micrograph images) captured by the image capture system 120 to obtain three- dimensional (3D) i reconstructions of particles in the sample 110. In general, the image processing system 130 processes the 2D micrograph images to generate a set of representative 2D projections of a type of particle (e.g., a macromolecule) from the 2D micrograph images, and reconstruct a 3D image of the type of particle using the set of representative 2D projections. As will be described in more detail below, the image processing system 130 can receive workflow configuration data 135 defining a workflow for performing operations of the process for generating the set of representative 2D projections and reconstructing the 3D image. For example, workflow configuration 135 can define one or more loops or repeated steps in the operations of generating the representative 2D projections or the 3D reconstruction, and / or define one or more algorithms or options used in the operations.

[0033] The image processing system 130 can output the 3D reconstruction 140 of the particle. For example, the system 130 can output the 3D reconstruction 140 in a particular file format that represents both the structural information. In some implementations, the system 130 can output the 3D reconstruction 140 to a display device, and uses 3D visualization tools that allow a user to visualize and explore the reconstruction 140 via the display device and user input devices.

[0034] The 3D reconstruction 140 can be used in a variety of scenarios and applications. For example, in structural analysis of macromolecules, the 3D reconstruction 140 provides detailed insights into the structure of biological macromolecules, such as proteins or viruses, at near-atomic resolution. Researchers can analyze the structural features, interactions, and overall architecture of the molecules. The 3D reconstruction can also be used in functional interpretation. Understanding the 3D structure helps elucidate the biological function of the molecules. It allows researchers to infer mechanisms of action, ligand binding sites, and conformational changes critical for function. In drug discovery, detailed structuralinformation can guide drug discovery efforts by identifying potential target sites for therapeutic intervention. It aids in designing drugs that interact with specific molecular structures. In biologies development, detailed 3D structural information can enable epitope mapping, mechanistic insights, and structure-guided protein engineering of protein / protein affinity tuning, pH dependent binding, and precise removal of common biologies development liabilities. In vaccine development, the 3D reconstruction of viral particles can contribute to vaccine development by providing insights into the viral structure. Understanding surface features helps in designing vaccines that elicit effective immune responses.

[0035] FIG. 2 is a flow diagram illustrating an example process 200 for determining a 3D structure of a type of particle in a sample. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, an image processing system, e.g., the image processing system 130 of FIG. 1, appropriately programmed in accordance with this disclosure, can perform the process 200. In a particular example, the image processing system 130 can be implemented by a cloudbased server performing the operations of the system 130.

[0036] As described with reference to FIG. 1, the sample can be any appropriate Cryo-EM sample, and can include particles such as protein molecules, viral particles, and cellular components such as organelles or macromolecular complexes. Without loss of generality, the process 200 is focused on determining the 3D structure of a particular particle type in the sample, such as a particular protein molecule, a particular viral particle, or a particular cellular component. In this Specification, the particular particle type for which the 3D reconstruction is generated is also termed the “target particle ty pe” or the “target particle”.

[0037] At 210, the system obtains 2D micrograph images of the sample, e.g., from a Cryo- EM image capture system. In some implementations, multiple images are captured for the same sample to maximize information and overcome the limitations of the Cryo-EM system. These images can vary' in defocus levels and viewing angles to enhance contrast and gather data from different perspectives. In some cases, a rapid sequence of images is captured as a video, allowing post-processing to mitigate motion artifacts.

[0038] In some implementations, at 215, the system receives workflow' configuration data that defines a workflow for performing operations for reconstructing the 3D image. As willbe described below, processing the 2D micrograph images to generate the 3D reconstruction of the particles includes multiple steps. The workflow configuration data can define the sequence of steps included in the workflow. The workflow configuration data can further define one or more loops or repeated steps in the operations, and / or or more algorithms or options used in the operations. In some cases, the configuration data includes one or more tags that mark the repeated steps. This tagging mechanism helps organize the program into distinct running sets, preventing repeated options from interfering with the outcomes of previous steps.

[0039] By using a pre-defined workflow configuration, users have the flexibility to define and arrange the workflow as needed. The system accepts pre-defined configurations that can be tailored to the specific requirements of the reconstruction process. The workflow can incorporate repeatable steps without limitations. Tags can be included in the workflow configuration to identify and segregate repeat steps, preventing them from conflicting with options in previous stages. This flexibility is particularly valuable for iterative processes or when specific steps need to be executed multiple times.

[0040] The pre-defined workflow can be executed automatically without requiring manual intervention. This automation streamlines the reconstruction process, reducing the need for constant monitoring and input from users. The absence of manual involvement allows the system to efficiently utilize computational resources. The automated workflow can proceed efficiently, making the most of idle time without relying on manual oversight.

[0041] Furthermore, users have the freedom to initiate the entire workflow at any time, e.g., by using a scheduled workflow. This flexibility contrasts with traditional approaches that might require careful planning, waiting for job completions, and setting up long runs during specific time periods, such as overnight or over weekends.

[0042] The workflow configuration can be based on expert experience. This would assist users, including those without specialized expertise in the field, to effectively utilize the system to reconstruct complex structures. This allows a broader user base to engage in Cryo- EM studies.

[0043] In some implementations, at 220, the system processes one or more temporal sequences of 2D micrograph images to correct motion artifacts. Motion-induced artifacts in Cryo-EM image data can be caused by factors such as beam-induced motions, specimen drift,stage instabilities, and thermal fluctuations. Any appropriate motion correction algorithms can be used to correct the motion artifact. For example, algorithms like MotionCor2 employ cross-correlation and iterative refinement to accurately track and correct translational and rotational displacements within individual frames of micrographs. In some cases, strategies such as dose weighting and frame-by-frame quality7assessment for further refinement can be used in motion correction.

[0044] In some implementations, at 230, the system estimates a contrast transfer function (CTF) of the 2D micrograph images and processes the 2D micrograph images to reduce blurring caused by the CTF. In Cryo-EM, the CTF characterizes how aberrations of the electron microscope modulates the features of the acquired images. The CTF is influenced by various factors, including the properties of the microscope, the sample thickness, and the defocus level. The CTF introduces blurring and phase reversals in the images, impacting the ability7to extract high-resolution information. Any appropriate processing can be applied to estimate and correct the CTF in the EM micrographs. For example, the system can estimate the CTF by analyzing the characteristic oscillations (Thon rings) present in the Fourier transform of the 2D micrograph images. These rings provide information about the defocus level and other parameters affecting the CTF. Once the CTF is estimated, the system can process the 2D micrograph images to correct for the blurring caused by the CTF. This correction can include, for example, applying an inverse filter to counteract the effects of the CTF. In some cases, CTF correction can be performed as an iterative process. After an initial estimation and correction, the images can undergo additional rounds of refinement to improve the accuracy of the CTF parameters and further enhance image quality7.

[0045] At 240, the system generates a set of representative 2D projections of the target particle from the 2D micrograph images. Examples of the process of generating the set of representative 2D projections of the target particle will be described in detail with reference to FIG. 3. In general, the goal of this process is to determine a set of high-quality 2D projections of the target particle at a variety of projection angles, so that the 2D projections can be used to reconstruction the 3D structure of the target particle. Each representative 2D projection is a cropped image containing a single instance of the 2D projection of the target particle at a particular angle. As will be discussed in more details with reference to FIG. 3, the representative 2D projection can be a selected lass average image, which is obtained byperforming an averaging across 2D particle projections in a respective class for improving the signal -to-noise ratio (SNR) of the representative 2D projection.

[0046] Although the 2D micrograph images obtained from the image capture device are 2D projection images of the sample, in general, each 2D micrograph image contains a large number of instances of 2D projections of the target particle at different angles. Furthermore, artifacts may be present in parts of the micrograph, potentially confusing with the target particles. The artifacts can be caused by contaminants, such as dust or other debris, as well as ice contamination which occurs in the vitrified ice layer that surrounds the sample. The artifacts can also be caused by out-of-focus particles, over-exposed or under-exposed areas, uneven illumination, charge build-up from electron charging, optical aberration, noises, and so on. In order to obtain a high-quality 3D reconstruction of the target particle, it is essential to obtain the representative 2D projections of the target particle that captures the structural information of the target particle with as little artifact as possible. As will be described with reference to FIG. 3, the process generally includes identifying 2D particle projections in the 2D micrograph images, classifying the plurality of 2D particle projections into a set of classes, determining a respective class representation image representing the 2D particle projections in each class, and selecting the set of representative 2D projections from the class representation images of the set of classes. In some cases, the process for generating the set of representative 2D projections is performed for a plurality of iterations. That is, in each iteration after the first iteration, the set of 2D particle projections in the 2D micrograph image are identified using templates based on the set of representative 2D projections selected by the preceding iteration.

[0047] At 250, the system reconstructs a 3D image of the target particle using the set of representative 2D projections. The system can use any appropriate algorithms to perform 3D reconstruction from the 2D projections, e.g., by using Fourier transforms and back-projection techniques. In some implementations, the system can use an iterative approach to refine the 3D reconstruction. For example, the system can generate an initial 3D reconstruction of the target particle, and perform an iterative process including re-projecting the 3D reconstruction into 2D space to generate simulated projections, compare the simulated projections with the representative 2D projections, and adjusting the 3D reconstruction to minimize discrepancies between simulations and observations.

[0048] As the process of generating representative 2D projections can be computationally demanding, especially when dealing with high-resolution micrograph images, a strategy can be employed to optimize efficiency. In some implementations, the system can perform 240 to generate initial representative 2D projections based on down-sampled images (e.g., images with reduced resolution), and then use the initial representative 2D projections as a guide to obtain high-resolution representative 2D projections for the 3D reconstruction. Downsampling involves reducing the resolution and number of pixels of the original high- resolution micrograph images. This significantly decreases the computational load for the steps of 240, such as particle identification, classification, averaging, and selection. By working with smaller image sizes, the system can quickly identify particle projections, expediting 240. The initial representative 2D projections obtained from the down-sampled images can be used to guide the selection of corresponding particles in the original high- resolution images. Once the high-resolution particles are picked, the system can generate high-resolution representative 2D projections (e.g., class averages) for the 3D reconstruction. This approach improves computational efficiency while maintaining reconstruction quality. The initial stages are accelerated by working with smaller images, while the final 3D reconstruction is performed based on the full-resolution data to ensure the highest quality 3D reconstruction.

[0049] At 260, the system outputs the 3D image of the target particle. In some implementations, the system can output the 3D reconstruction in a particular file format that represents the structural information and its associated metadata of the 3D reconstruction. In some implementations, the system can output the 3D reconstruction to a display device, and uses 3D visualization tools to allow a user to visualize and explore the reconstruction via the display device and user input devices.

[0050] FIG. 3 is a flow diagram illustrating an example of the sub-process 240 for generating the set of representative 2D projections of the target particle from the 2D micrograph images. For convenience, the sub-process 240 will be described as being performed by a system of one or more computers located in one or more locations. For example, an image processing system, e.g., the image processing system 130 of FIG. 1, appropriately programmed in accordance with this disclosure, can perform the process 240.

[0051] At 310, the system identifies a plurality of 2D particle projections in the 2D micrograph images. Each 2D particle projection is a corresponding cropped portion of oneof the 2D micrograph images that depicts a single instance of a 2D projection of the target particle at a certain angle. To identify a 2D particle projection, the system can first identify a location (e.g., the coordinates of the center) of the 2D particle projection in the corresponding 2D micrograph image, and then extract the 2D particle projection by cropping the portion of the 2D micrograph image that corresponds to a region of interest (ROI) of the 2D particle projection. The system can use any suitable algorithm for identifying the 2D particle projections. For example, the system can use a template-based algorithm. Alternatively or in addition, the system can use a template-free algorithm.

[0052] Template-based algorithms rely on predefined templates or reference images of the target particles. These templates can be generated based on prior knowledge of the particle's appearance and structure, user input, and / or have been generated earlier in the processing pipeline. The algorithms compare the micrograph with the templates to find regions that closely match the expected particle shape and features. This matching process can be performed using techniques like cross-correlation, template matching, or machine learningbased approaches.

[0053] Template-free algorithms do not rely on predefined templates. Instead, they employ techniques to identify particles based on inherent features, patterns, and statistical properties within the micrographs. For example, these algorithms can use methods such as local variance analysis, edge detection, or machine learning to identify regions that deviate from the background noise. Machine learning approaches may involve training a model to distinguish between particles and noise.

[0054] At 320, the system classifies the plurality of 2D particle projections into a set of classes. The goal of the classification is to group particle projections at the same projection (or similar) projection angles. The system can use any appropriate techniques to perform the particle projection classification, including, for example, principal component analysis (PCA), multivariate statistical analysis, template matching, feature extraction-based techniques, and / or supervised or unsupervised machine learning.

[0055] At 330. the system determines, for each class in the set of classes, a respective class representation image representing the 2D particle projections in the respective class. For example, the class representation image can be a class average image, which is obtained by performing an averaging across the 2D particle projections in the respective class. Byperforming in-class averaging, an improved signal-to-noise ratio (SNR) can be reached for the class average image compared to the individual images in the class.

[0056] At 340, the system obtains metadata for each class representation image and uses the metadata to select the set of representative 2D projections from the class representation images of the set of classes. The metadata can include one or more of (i) an average pixel value of the class representation image, (ii) a maximum pixel value of the class representation image, (iii) a minimum pixel value of the class representation image, (iv) a pixel value standard deviation of the class representation image, (v) a pixel value signal-to-noise ratio (SNR) of the class representation image, and / or (vi) a class distribution value characterizing a number of 2D particle projections in the class for the respective class representation image. In this Specification, the pixel value refers to a value assigned to a pixel in an image, such as an intensity value of the pixel. In some cases, the SNR is calculated as a ratio of the maximum pixel value to the minimal pixel value, that is, SNR — Imax / lmin.

[0057] In some implementations, to select the set of representative 2D projections from the class representation images, the system can exclude a class representation image from the selection if one or more parameter values in the metadata of the class representation image fall outside of a particular range. That is. for each respective class, the system determines whether each of the one or more parameter values is within a respective range (e.g., defined by one or more respective threshold values for the parameter value). If each of the one or more parameter values is within the respective range, the system includes the respective class representation image in the selection. On the other hand, if at least one of the parameter values is not within the particular range, the system excludes the respective class representation image from the selection. As an example, if the average pixel value of a class representation image is out of a particular range, or if the SNR of the class representation image is out of a particular range, the system can exclude the respective class representation image from the selection. In some cases, for each respective parameter value, the respective range is determined based on a predefined quantile of the parameter values for the set of class representation images. For example, for the average pixel value, the particular range can be set at 90% of the average pixel values of the set of class representation images. That is, if the average pixel value of a particular class representation image is within the 10% highest of all the class representation images, the particular class representation image is considered an outlier and excluded from the selected set of representative 2D projections.

[0058] FIG. 4 shows examples of metadata of a set of representative 2D particle projections. The metadata includes the average pixel value, the pixel value standard deviation the maximum pixel value, the minimum pixel value, the pixel value SNR, and the class distribution value.

[0059] Referring back to FIG. 3, in some implementations, to select the set of representative 2D projections from the class representation images using the metadata (at 340), the system can use a machine-learning model to process the metadata to generate a prediction output for selecting the representative 2D projections. For each respective class representation image, the system can generate a respective model input for the respective class representation image using the respective metadata, process the respective model input using the machine-learning model to generate the respective output, and determine whether to include the respective class representation image in the selection based on the output. The machine-learning model can include any appropriate predictive machine-learning model, such as a neural network and / or a random forest model. The input to the machine-learning model can be a vector including parameters in the metadata, such as the average pixel value, the maximum pixel value, the minimum pixel value, the pixel value standard deviation, the pixel value SNR, and a class distribution value of a respective class representation image. In some implementations, the output of the machine-learning model can be a classification indicating whether the respective class representation image should be included in the selection. In some implementations, the output of the machine-learning model can be a prediction score predicting the likelihood of the respective class representation image belonging to the selection. In a particular example, the machine-learning model is a random forest model that includes a plurality of decision trees. During training, each tree is trained independently on a random subset of the training data. The training process involves randomly sampling instances with replacement, which introduces diversity among the trees and helps to reduce overfitting. During inference, the predictions of individual trees are combined through a majority voting mechanism. The class that receives the most votes across all trees is the final predicted class.

[0060] In some implementations, the process including steps 310-340 can be performed for a plurality of iterations. In each iteration after the first iteration, the system identifies the set of 2D particle projections in the 2D micrograph image (at 310) using templates based on the set of representative 2D projections selected by the preceding iteration. That is, after thesystem performs 340 for an iteration, the system determines whether to continue the iteration (at 350). If the system determines to continue the iteration, the system uses the representative 2D projections as particle templates (at 370), and continues to the next iteration in which the system identifies the 2D particle projections in the 2D micrograph image (at 310) using templates based on the set of representative 2D projections selected by the preceding iteration. When a stop condition is satisfied at 350, the system finishes updating the representative 2D projections at 360. The stop condition can include (i) that a predefined number of iterations have been performed, (ii) that the changes to the updated representative 2D projections are within a certain threshold, or (iii) one or more quality parameters (e.g., the SNR) of all updated representative 2D projections are within a predefined range. This iterative process can repeatedly refine the representative 2D projections to improve their quality. The iteration process can be pre-defined, e.g., in the workflow configuration data.

[0061] In some other implementations, the process including steps 320-340 can be performed for a plurality of iterations. In each iteration, the system performs the initial classification of 2D particle projections (320), generates class representation images (330), obtains metadata for the class representation images, and selects representative 2D projections using the metadata (340), e.g., determines whether to select a representative 2D projection based on the determination whether one or more parameter values of the metadata of the corresponding class representation image is within a range as defined by one or more respective threshold values for the parameter values. At the end of the iteration, the system determines whether to continue the iteration (at 350). If yes, the system continues to the next iteration in which the system performs the classification again (at 320) on the 2D particle projections that correspond to the set of representative 2D projections that have been selected (at 340) in the preceding iteration. When the stop condition is satisfied at 350, the system finishes updating the representative 2D projections at 360. This iterative process can repeatedly refine the representative 2D projections to improve their quality. This iterative process can repeatedly refine the representative 2D projections to improve their quality. The iteration process can be pre-defined, e.g., in the workflow configuration data.

[0062] In some cases, the system may adjust the threshold values across the iterations used for selecting representative projections. As an example, at 340 of an initial iteration, the system can select a class representation image as one of the selected representative 2D projections if the SNR of the class representation image is above a respective SNR thresholdfor the initial iteration. As the system progresses to the next iteration, the system can adjust the SNR threshold (e.g., with a predefined increment) for selecting the representative 2D projections at 340 of that next iteration. That is, at the initial iteration, the system can use a lower threshold for the SNR to include a wider range of projections. As iterations progress, this threshold can be adjusted incrementally, making the selection criteria more stringent and focusing on projections with increasingly higher SNR values. This approach allows for a more nuanced refinement process, starting with a broader selection and gradually narrowing it down to the representative projections with the best quality. While the above example focuses on using the SNR for selecting the representative 2D projections, the system can utilize other metadata parameters for the selection. These could include resolution estimation or quality scores predicted by Al models, or any other relevant metric that can assess the quality and representativeness of a projection.

[0063] FIG. 5 is a block diagram of an example computer system 500 that can be used to perform the operations described above. The system 500 includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of the components 510, 520, 530, and 540 can be interconnected, for example, using a system bus 550. The processor 510 is capable of processing instructions for execution within the system 500. In one implementation, the processor 510 is a single-threaded processor. In another implementation, the processor 510 is a multi -threaded processor. The processor 510 is capable of processing instructions stored in the memory 520 or on the storage device 530.

[0064] The memory 520 stores information within the system 500. In one implementation, the memory 520 is a computer-readable medium. In one implementation, the memory 520 is a volatile memory unit. In another implementation, the memory 520 is a non-volatile memory7unit.

[0065] The storage device 530 is capable of providing mass storage for the system 500. In one implementation, the storage device 530 is a computer-readable medium. Tn various different implementations, the storage device 530 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (for example, a cloud storage device), or some other large capacity storage device.

[0066] The input / output device 540 provides input / output operations for the system 500. In one implementation, the input / output device 540 can include one or more network interface devices, for example, an Ethernet card, a serial communication device, for example, a RS- 232 port, and / or a wireless interface device, for example, a 502.11 card. In another implementation, the input / output device can include driver devices configured to receive data and send output data to other input / output devices, for example, keyboard, printer and display devices 760. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0067] Although an example processing system has been described in FIG. 5, implementations of the subject matter and the functional operations described in this disclosure can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this disclosure and their structural equivalents, or in combinations of one or more of them.

[0068] This disclosure uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this disclosure can be implemented in digital electronic circuitry, in tangibly- embodied computer software or firmware, in computer hardware, including the structures disclosed in this disclosure and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this disclosure can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g.. a machine-generated electrical, optical, or electromagnetic signal, that is generated to encodeinformation for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0069] The term "data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0070] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0071] In this disclosure, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0072] Similarly, in this disclosure the term “engine” is used broadly to refer to a software- based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules orcomponents, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0073] The processes and logic flows described in this disclosure can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry' and one or more programmed computers.

[0074] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry'. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g.. magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g.. a universal serial bus (USB) flash drive, to name just a few.

[0075] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory', media and memory' devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0076] To provide for interaction with a user, embodiments of the subject matter described in this disclosure can be implemented on a computer having a display device, e.g.. a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interactionwith a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0077] Data processing apparatus for implementing machine-learning models can also include, for example, special -purpose hardware accelerator units for processing common and compute-intensive parts of machine-learning training or production, i.e., inference, workloads.

[0078] Machine learning models can be implemented and deployed using a machine-learning framework, e.g., a PyTorch or a TensorFlow framework.

[0079] Embodiments of the subject matter described in this disclosure can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this disclosure, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g.. the Internet.

[0080] The computing system can include clients and servers. A client and server are generally remote from each other and ty pically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, whichacts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0081] While this disclosure contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this disclosure in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0082] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

Claims

CLAIMS1. A computer-implemented method for determining a three-dimensional (3D) structure of a ty pe of particle in a sample, the method comprising: obtaining one or more two-dimensional (2D) micrograph images of the sample; performing a process for generating a set of representative 2D projections of the ty pe of particle from the 2D micrograph images, the process comprising: identifying a plurality of 2D particle projections in the 2D micrograph images; classifying the plurality of 2D particle projections into a set of classes; determining, for each class in the set of classes, a respective class representation image representing the 2D particle projections in the respective class; obtaining respective metadata for each class representation image, the respective metadata comprising one or more of: an average pixel value, a maximum pixel value, a minimum pixel value, a pixel value standard deviation, or a pixel value signal-to- noise ratio (SNR) of pixels in the class representation image; and selecting, using the respective metadata for each class representation image, the set of representative 2D projections from the class representation images of the set of classes; reconstructing a 3D image of the type of particle using the set of representative 2D projections; and outputting the 3D image of the ty pe of particle.

2. The method of claim 1. wherein determining the class representation image representing the 2D particle projections in the respective class comprises: performing an averaging across the 2D particle projections in the respective class to obtain an averaged image as the class representation image.

3. The method of claim 1 or claim 2, wherein the process for generating the set of representative 2D projections is performed for a plurality of iterations, wherein in each iteration after the first iteration, the set of 2D particle projections in the 2D micrograph image are identified using templates based on the set of representative 2D projections selected by the preceding iteration.

4. The method of any preceding claim, wherein the respective metadata further comprises a class distribution value characterizing a number of 2D particle projections in the class for the respective class representation image.

5. The method of any preceding claim, wherein selecting the set of representative 2D projections from the class representation images of the set of classes comprises, for each respective class: determining one or more parameter values from the respective metadata of the respective class representation image; determining whether each of the one or more parameter values is within a respective range defined by one or more respective threshold values for the parameter value; in response to determining that each of the one or more parameter values is within the respective range, including the respective class representation image in the selection; and in response to determining that at least one of the parameter values is not within the respective range, excluding the respective class representation image from the selection.

6. The method of claim 5, wherein the process for generating the set of representative 2D projections is performed for a plurality of iterations, wherein in each iteration after the first iteration, classifying the plurality of 2D particle projections is performed on a subset of 2D particle projections that correspond to the set of representative 2D projections having been selected in the preceding iteration.

7. The method of claim 6, wherein at least one of the one or more respective threshold values is adjusted across the plurality of iterations.

8. The method of any of claims 5-7, wherein for each respective parameter value, the respective range is determined based on a respective predefined quantile of the respective parameter value.

9. The method of any of claims 5-8. wherein the one or more parameter values used for selecting the set of representative 2D projections comprise the average pixel value and the pixel value signal-to-noise ratio.

10. The method of claim 9, wherein the pixel value signal-to-noise ratio is computed as a ratio of the maximum pixel value to the minimal pixel value.

11. The method of any preceding claim, wherein selecting, using the respective metadata for each respective class representation image, the set of representative 2D projections from the class representation images of the set of classes comprises, for each respective class representation image: generating a respective model input for the respective class representation image using the respective metadata; processing the respective model input using a machine-learning model to generate respective output; and determining whether to include the respective class representation image in the selection based on the output.

12. The method of claim 11. wherein generating the respective model input comprises: generating a respective vector using the average pixel value, the maximum pixel value, the minimum pixel value, the pixel value standard deviation, or the pixel value SNR, and a class distribution value; and generating the respective model input from the respective vector.

13. The method of any preceding claim, wherein the 2D micrograph images comprise electron microscope (EM) images of the sample.

14. The method of claim 13, wherein the 2D micrograph images comprise a set of EM images of the sample taken from different angles.

15. The method of any preceding claim, further comprising: before performing the process for generating the set of representative 2D projections, processing the 2D micrograph images to correct motion artifacts.

16. The method of any preceding claim, further comprising:before performing the process for generating the set of representative 2D projections, estimating a contrast transfer function (CTF) of the 2D micrograph images and processing the 2D micrograph images to reduce blurring caused by the CTF.

17. The method of any preceding claim, further comprising: receiving configuration data defining a workflow for performing operations of the process for generating the set of representative 2D projections, and reconstructing the 3D image using the set of representative 2D projections, and performing the operations based on the workflow defined in the configuration data.

18. The method of claim 17, wherein the configuration data defines one or more loops or repeated steps in the operations.

19. The method of claim 18, wherein the configuration data defines one or more algorithms or options used in the operations.

20. The method of claim 19, wherein the configuration data includes one or more tags that mark the repeated steps.

21. A system comprising: one or more computers; and one or more storage devices storing instructions that when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-20.

22. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-20.

Images