Automated image processing for electron microscopy

By identifying and classifying 2D particle projections, utilizing metadata and machine learning models to select representative projections, and optimizing the workflow configuration of Cryo-EM, the problems of artifacts and high computational resources in Cryo-EM are solved, achieving efficient and accurate 3D particle reconstruction and resource utilization.

CN123125619APending Publication Date: 2026-11-17SANOFI SA(FR)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202580028556.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-08-27
Filing Date
2025-02-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing Cryo-EM technology suffers from artifacts and noise in EM micrographs, making it difficult to accurately reconstruct high-quality 3D particle structures. It also has high computational resource requirements and lacks automation and flexibility.

Method used

By identifying and classifying 2D particle projections, representative projections are selected using metadata such as average pixel value and signal-to-noise ratio. Combined with machine learning models, workflow configuration data is optimized to automate the reconstruction process, reducing artifact effects and computational resource requirements.

Benefits of technology

It improves the accuracy and efficiency of 3D particle reconstruction, reduces the demand for computing resources, expands the applicability to the user group, and enhances the system's flexibility and the utilization rate of computing resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125619A_ABST
    Figure CN123125619A_ABST
Patent Text Reader

Abstract

Methods, computer systems, and apparatus for determining the 3D structure of a class of particles, including computer programs encoded on a computer storage medium. The system acquires one or more two-dimensional (2D) micrographs of the sample and performs a process for generating a set of representative 2D projections of the class of particles from these 2D micrographs. The system acquires metadata for each class-representation image and uses this metadata to select a set of representative 2D projections from these class-representation images. The system reconstructs a 3D image of the class of particles using the set of representative 2D projections and outputs a 3D image of the class of particles.
Need to check novelty before this filing date? Find Prior Art

Description

Cross-references to related applications

[0001] This application claims priority to EP application No. 24315049.7, filed February 20, 2024; U.S. Provisional Patent Application No. 63 / 663,224, filed June 24, 2024; and EP application No. 24315390.5, filed August 27, 2024, the disclosures of all of which are hereby incorporated in their entirety by reference. Technical Field

[0002] The present invention generally relates to systems and methods for automating image processing for reconstructing three-dimensional (3D) particle images from an electron microscope (EM), and more specifically to identifying two-dimensional (2D) projections of particles in EM images for 3D reconstruction. Background Technology

[0003] Cryo-electron microscopy (Cryo-EM) involves rapidly freezing samples to preserve their native conformation, thus overcoming structural distortions caused by conventional fixation methods. Using electron microscopy, Cryo-EM captures high-resolution images of these rapidly frozen samples, and subsequent computational analysis, including single-particle reconstruction, helps generate three-dimensional structures with near-atomic resolution.

[0004] Compared to other structural biology techniques such as X-ray crystallography, Cryo-EM does not require the preparation of crystals, making it particularly advantageous for studying flexible molecules and large protein complexes. Furthermore, Cryo-EM can capture multiple conformations of molecules in a single experiment, providing valuable insights into their flexibility and function. Summary of the Invention

[0005] This disclosure describes methods, computer systems, and apparatus for determining the three-dimensional (3D) structure of a class of particles in a sample, including computer programs encoded on a computer storage medium.

[0006] In one aspect, this disclosure provides a method for determining the 3D structure of a class of particles. This method can be implemented by a system comprising one or more computers. The system acquires one or more two-dimensional (2D) micrographs of the sample and performs a process for generating a set of representative 2D projections of the class of particles from these 2D micrographs. To generate the set of representative 2D projections, the system identifies multiple 2D particle projections in the 2D micrographs, classifies these multiple 2D particle projections into a set of categories, and for each category, determines a corresponding category representation image representing these 2D particle projections in that category; obtains corresponding metadata for each category representation image, the corresponding metadata including one or more of the following: the average pixel value, maximum pixel value, minimum pixel value, standard deviation of pixel values, or signal-to-noise ratio (SNR) of pixel values ​​in the category representation image; and uses the corresponding metadata for each category representation image to select the set of representative 2D projections from these category representation images of the group of categories. The system reconstructs a 3D image of the class of particles using the set of representative 2D projections and outputs a 3D image of the class of particles.

[0007] In some implementations, in order to determine the category representation image representing these 2D particle projections in the corresponding category, the system averages these multiple 2D particle projections in the corresponding category to obtain an average image as the category representation image.

[0008] In some implementations, the process for generating the set of representative 2D projections is performed in multiple iterations. In each iteration after the first iteration, the set of 2D grain projections in the 2D photomicrograph is identified using a template based on the set of representative 2D projections selected by the previous iteration.

[0009] In some implementations, the corresponding metadata further includes a category distribution value that characterizes the number of 2D grain projections in that category of the image represented by the corresponding category.

[0010] In some implementations, to select a group of representative 2D projections from these category representation images of the group of categories, for each corresponding category, the system determines one or more parameter values ​​from the corresponding metadata of the corresponding category representation image, determining whether each of the one or more parameter values ​​is within a corresponding range defined by one or more corresponding thresholds of the parameter value. In response to determining that each of the one or more parameter values ​​is within the corresponding range, the system includes the corresponding category representation image in the selection. In response to determining that at least one of the parameter values ​​is not within the corresponding range, the system excludes the corresponding category representation image from the selection. In some cases, for each corresponding parameter value, the corresponding range is determined based on a corresponding predefined quantile of the corresponding parameter value. In some cases, the one or more parameter values ​​used to select the group of representative 2D projections include the average pixel value and the pixel value signal-to-noise ratio. In some cases, the pixel value signal-to-noise ratio is calculated as the ratio of the maximum pixel value to the minimum pixel value.

[0011] In some implementations, to select a representative 2D projection from these category representation images of the group of categories, for each corresponding category representation image, the system uses the corresponding metadata to generate a corresponding model input for that category representation image, processes the corresponding model input using a machine learning model to generate a corresponding output, and determines whether to include the corresponding category representation image in the selection based on the output. For example, to generate the corresponding model input, the system can use the average pixel value, the maximum pixel value, the minimum pixel value, the standard deviation of the pixel value, or the pixel value SNR and the category distribution value to generate a corresponding vector, and generate the corresponding model input from the corresponding vector.

[0012] In some implementations, these 2D photomicrographs include electron microscope (EM) images of the sample. In some cases, these 2D photomicrographs include a set of EM images of the sample taken from different angles. In some cases, the system processes these 2D photomicrographs to correct for motion artifacts before performing the process for generating the set of representative 2D projections. In some cases, the system estimates the contrast transfer function (CTF) of these 2D photomicrographs and processes them to reduce blurring caused by the CTF before performing the process for generating the set of representative 2D projections.

[0013] In some implementations, the system further receives configuration data that defines a workflow for performing the operations of generating the set of representative 2D projections; reconstructing the 3D image using the set of representative 2D projections; and performing these operations based on the workflow defined in the configuration data. In some cases, the configuration data defines one or more looping or repeating steps in these operations. In some cases, the configuration data defines one or more algorithms or options used in these operations. In some cases, the configuration data includes one or more labels marking these repeating steps.

[0014] This disclosure also provides a system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the methods described above.

[0015] This disclosure also provides one or more computer storage media that store instructions that, when executed by one or more computers, cause the one or more computers to perform the methods described above.

[0016] The subject matter described in this disclosure may be implemented in particular embodiments to achieve one or more advantages.

[0017] Cryo-EM has become a key technology in multiple fields, including structural biology, drug development, vaccine development, and materials science. It provides detailed structural information about proteins and viruses, significantly advancing our understanding of their roles in health and disease. These structural insights derived from Cryo-EM not only guide the precise targeting of new drugs but also aid vaccine development by elucidating viral structures. Furthermore, the exploration of biomolecular structures has inspired the design of novel materials with unique properties, thus expanding the impact of Cryo-EM across multiple scientific disciplines.

[0018] Several challenges exist in conventional techniques for processing EM micrographs to reconstruct 3D particle structures. EM micrographs received from image capture devices contain artifacts caused by contaminants such as dust or other debris, as well as ice contamination from the vitrified ice layer surrounding the sample. Artifacts can also be caused by defocused particles, overexposed or underexposed areas, uneven illumination, charge buildup due to electron charging, optical aberrations, noise, etc. To obtain high-quality 3D reconstructions of target particles, it is crucial to obtain representative 2D projections of the target particles that capture the structural information of the target particles with as few artifacts as possible. This process typically involves selecting representative 2D projections of the target particles from a set of candidate 2D projections. This specification provides techniques for improving this selection process. Instead of relying solely on image features for selection, the system extracted metadata, such as average pixel values ​​and signal-to-noise ratio (SNR), from the image for each category representation and selected representative projections based on the metadata, using (i) exclusion criteria to filter out projections with outliers, or (ii) machine learning models (e.g., random forests) to predict which projections best represent the target particles.

[0019] Metadata-based projection selection offers advantages over relying solely on image features. This is because image features can be variable and sensitive to factors including (i) image scale—changing the image size can significantly alter pixel-level features, (ii) image normalization—adjusting the range of pixel intensity can affect feature values, and (iii) positional variations—particle offset within the image affects feature extraction. Metadata parameters are more robust to these variations. They capture fundamental image properties such as average pixel values, signal-to-noise ratio, and class distribution. These properties are less sensitive to slight image manipulation or changes in particle position. Therefore, metadata-based selection provides a more stable and reliable basis for selecting representative particle projections and reduces the risk of selecting projections based on features that may be artifacts from image processing or particle localization, resulting in more accurate and robust 3D reconstruction. This more efficient selection process also reduces the number of iterations required to obtain high-quality particle reconstructions, thereby reducing computational resource requirements and increasing the speed of the computation process without requiring manual input from expert users.

[0020] Additionally, in some implementations, the provided system uses user-defined workflow configuration data to automate the reconstruction workflow. By using user-defined workflow configuration data, users can define and arrange the sequence, loops, algorithms, and options of steps as needed to customize the workflow to specific reconstruction requirements. Workflows can contain repeatable steps without restriction, enabling iterative processes and optimizations. Workflow configuration data can use tags to organize repeating steps to avoid conflicts. Predefined workflows can be executed automatically, reducing manual intervention and maximizing computational resource utilization, thereby reducing computational resource requirements and increasing the speed of the computation process. Users can start workflows at any time without careful planning around job completion. Workflow configurations can be based on expert knowledge, enabling users without extensive experience to effectively reconstruct complex structures. Therefore, user-defined workflow configurations improve efficiency and productivity through automation, enhance the flexibility of adapting workflows to different projects and requirements, optimize the use of computational resources, and expand the accessibility of Cryo-EM research to a wider user base.

[0021] Details of one or more embodiments of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0022] Figure 1 An example environment for reconstructing 3D images of Cryo-EM samples is shown.

[0023] Figure 2 This is a flowchart of an example process for reconstructing 3D images from photomicrographs.

[0024] Figure 3 This is a flowchart of an example process for generating a representative 2D projection.

[0025] Figure 4 An example of a representative 2D particle projection and its metadata is shown.

[0026] Figure 5 This is a block diagram of an example computer system.

[0027] Similar reference numerals and names in the various figures indicate similar elements. Detailed Implementation

[0028] Figure 1 An example environment 100 is shown for reconstructing a 3D image 140 of a sample 110 using an image processing system 130.

[0029] Sample 110 can be any suitable sample and may include, for example, purified protein molecules, membrane proteins, viral particles, and cellular components (such as organelles or macromolecular complexes). Different types of samples can be prepared using sample-specific processes. For example, purified protein molecules can be concentrated and stabilized to optimize particle density. Membrane proteins can be detergent-solubilized and incorporated into nanophospholipid discs or amphiphilic polymers. Viral particles typically require purification from host cells, often using gradient or affinity chromatography. Cellular components such as organelles or macromolecular complexes can be processed using homogenization and fractionation techniques. Sample 110 is vitrified, which involves rapid freezing to liquid ethane or propane to maintain natural hydration and prevent ice crystal formation for high-resolution studies. Sample 110 can be placed on a cryogrid. The cryogrid is a support platform used to hold and position sample 100.

[0030] Image acquisition system 120 includes an electron microscope (EM). During image acquisition, the EM uses a focused electron beam to scan the sample 110 placed on a cryogenic grid, thereby inducing elastic scattering of electrons. The scattered electrons are focused within the electron microscope through a series of magnetic lenses and projected onto a high-resolution detector, thus generating a two-dimensional (2D) projected image of the sample 110. Multiple images are typically captured with varying defocus amounts and from different perspectives. These perspectives are attributed to the rotational distribution of particles within the sample. In some cases, multiple images are captured at a high frame rate, allowing post-processing to mitigate motion caused by the electron beam. In the illustrative example, the detector acquires each image at 320 fps. In post-processing, each motion-corrected image is processed from a large number of sub-images.

[0031] Theoretically, the resolution of a Cryo-EM projection image is limited by the wavelength of the electron beam, which is much smaller than the typical size of biomolecules. For example, at a typical accelerating voltage of 200 kV, the electron wavelength is approximately 0.037 nm. In practice, several factors can reduce the resolution of a single projection image, such as electron beam-induced motion, defocusing, detector noise, sample heterogeneity, and large particle counts.

[0032] Image processing system 130 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which image processing methods can be implemented. (See reference...) Figure 2 and Figure 3In more detail, image processing system 130 processes 2D projected images (i.e., 2D photomicrographs) captured by image capture system 120 to obtain a three-dimensional (3D) reconstruction of the particles in sample 110. Typically, image processing system 130 processes 2D photomicrographs to generate a set of representative 2D projections of a class of particles (e.g., macromolecules) from the 2D photomicrographs and uses this set of representative 2D projections to reconstruct a 3D image of that class of particles. As will be described in more detail below, image processing system 130 may receive workflow configuration data 135 that defines a workflow for performing the operations used to generate the set of representative 2D projections and reconstruct the 3D image. For example, workflow configuration 135 may define one or more looping or repeating steps in the operation of generating representative 2D projections or 3D reconstruction, and / or define one or more algorithms or options used in the operation.

[0033] Image processing system 130 can output a 3D reconstruction 140 of the particles. For example, system 130 can output the 3D reconstruction 140 in a specific file format representing both structural information and other structural information. In some embodiments, system 130 can output the 3D reconstruction 140 to a display device and use a 3D visualization tool that allows users to visualize and explore the reconstruction 140 via the display device and user input device.

[0034] 3D Reconstruction 140 can be used in a variety of scenarios and applications. For example, in macromolecular structure analysis, 3D Reconstruction 140 provides detailed insights into the structure of biological macromolecules (such as proteins or viruses) at near-atomic resolution. Researchers can analyze the structural features, interactions, and overall architecture of molecules. 3D reconstruction can also be used for functional interpretation. Understanding 3D structures helps elucidate the biological functions of molecules. It allows researchers to infer mechanisms of action, ligand binding sites, and conformational changes that are crucial to function. In drug discovery, detailed structural information can guide drug discovery efforts by identifying potential targets for therapeutic interventions. It helps in designing drugs that interact with specific molecular structures. In biologics development, detailed 3D structural information enables epitope mapping, mechanistic insights, and structure-guided protein engineering that regulates protein / protein affinity, pH-dependent binding, and precisely removes common defects in biologics development. In vaccine development, 3D reconstruction of viral particles can facilitate vaccine development by providing insights into viral structure. Understanding surface features helps in designing vaccines that elicit an effective immune response.

[0035] Figure 2This is a flowchart illustrating an example process 200 for determining the 3D structure of a class of particles in a sample. For convenience, process 200 will be described as being performed by a system consisting of one or more computers located in one or more locations. For example, an image processing system appropriately programmed according to this disclosure (e.g., Figure 1 The image processing system 130 can execute process 200. In a particular example, the image processing system 130 may be implemented by a cloud-based server that performs the operations of system 130.

[0036] For reference Figure 1 As described, the sample can be any suitable Cryo-EM sample and can include particles such as protein molecules, viral particles, and cellular components (such as organelles or macromolecular complexes). Without loss of generality, process 200 focuses on determining the 3D structure of a specific particle type (such as a specific protein molecule, a specific viral particle, or a specific cellular component) in the sample. In this specification, the specific particle type for which a 3D reconstruction is generated is also referred to as the "target particle type" or "target particle".

[0037] At 210, the system acquires a 2D photomicrograph of the sample, for example, from a Cryo-EM image capture system. In some implementations, multiple images are captured for the same sample to maximize information and overcome the limitations of the Cryo-EM system. These images may vary in defocus and angle of view to enhance contrast and collect data from different angles. In some cases, a rapid sequence of images is captured as video, allowing for post-processing to mitigate motion artifacts.

[0038] In some implementations, at point 215, the system receives workflow configuration data that defines a workflow for performing operations to reconstruct the 3D image. As will be described below, processing 2D photomicrographs to generate 3D grain reconstruction involves multiple steps. The workflow configuration data may define the order of the steps included in the workflow. The workflow configuration data may further define one or more cyclic or repetitive steps in the operation, and / or one or more algorithms or options used in the operation. In some cases, the configuration data includes one or more labels marking these repetitive steps. This labeling mechanism helps organize the program into different sets of runs, thereby preventing repetitive options from interfering with the results of previous steps.

[0039] By using predefined workflow configurations, users can flexibly define and schedule workflows as needed. The system accepts predefined configurations that can be customized for the specific requirements of the rebuild process. Workflows can contain repeatable steps without restriction. Tags can be included in the workflow configuration to identify and separate repeating steps, thus preventing them from conflicting with options in previous stages. This flexibility is particularly valuable for iterative processes or when a particular step needs to be executed multiple times.

[0040] Predefined workflows can be executed automatically without manual intervention. This automation simplifies the rebuilding process and reduces the need for continuous user monitoring and input. The absence of manual intervention allows the system to utilize computing resources efficiently. Automated workflows can operate efficiently, making full use of idle time without relying on manual supervision.

[0041] Furthermore, users can freely initiate an entire workflow at any time, for example, by using a scheduled workflow. This flexibility contrasts with traditional methods that may require careful planning, waiting for tasks to complete, and setting up long runs during specific time periods (such as nights or weekends).

[0042] Workflow configuration can be based on expert experience. This will help users (including those without expertise in the field) effectively utilize the system to reconstruct complex structures. This allows for broader user participation in Cryo-EM studies.

[0043] In some implementations, at 220, the system processes one or more time series of 2D photomicrographs to correct for motion artifacts. Motion-induced artifacts in Cryo-EM image data can be caused by factors such as electron beam motion, sample drift, stage instability, and thermal fluctuations. Any suitable motion correction algorithm can be used to correct motion artifacts. For example, algorithms such as MotionCor2 employ cross-correlation and iterative refinement to accurately track and correct translational and rotational displacements within individual frames of the photomicrograph. In some cases, strategies such as dose weighting and frame-by-frame quality assessment can be used for further refinement in motion correction.

[0044] In some implementations, at 230, the system estimates the contrast transfer function (CTF) of the 2D photomicrographs and processes these 2D photomicrographs to reduce blurring caused by the CTF. In Cryo-EM, the CTF characterizes how aberrations of the electron microscope modulate the features of the acquired image. The CTF is affected by a variety of factors, including microscope characteristics, sample thickness, and defocusing. The CTF introduces blurring and phase inversion into the image, affecting the ability to extract high-resolution information. Any appropriate processing can be applied to estimate and correct the CTF in EM photomicrographs. For example, the system can estimate the CTF by analyzing characteristic oscillations (Thorn rings) present in the Fourier transform of the 2D photomicrographs. These rings provide information about the defocusing and other parameters affecting the CTF. Once the CTF is estimated, the system can process the 2D photomicrographs to correct the blurring caused by the CTF. This correction can include, for example, applying an inverse filter to counteract the effects of the CTF. In some cases, CTF correction can be performed as an iterative process. After initial estimation and correction, the image can undergo additional refinement rounds to improve the accuracy of the CTF parameters and further enhance image quality.

[0045] At position 240, the system generates a set of representative 2D projections of the target particle from the 2D photomicrograph image. An example of the process for generating this set of representative 2D projections of the target particle will be referenced. Figure 3 Detailed Description. Typically, the goal of this process is to determine a set of high-quality 2D projections of the target particle at multiple projection angles, so that these 2D projections can be used to reconstruct the 3D structure of the target particle. Each representative 2D projection is a cropped image containing a single instance of the target particle's 2D projection at a specific angle. (See reference...) Figure 3 In more detail, the representative 2D projection can be a selected category-averaged image, which is obtained by averaging multiple 2D particle projections in the corresponding category to improve the signal-to-noise ratio (SNR) of the representative 2D projection.

[0046] Although 2D photomicrographs obtained from image capture devices are 2D projections of the sample, each photomicrograph typically contains numerous instances of the target particle's 2D projection at different angles. Furthermore, artifacts may be present in multiple parts of the photomicrograph and can be confused with the target particle. Artifacts can be caused by contaminants (such as dust or other debris) and ice contamination from the vitrified ice layer surrounding the sample. Artifacts can also be caused by defocused particles, overexposed or underexposed areas, uneven illumination, charge buildup due to electron charging, optical aberrations, noise, etc. To obtain high-quality 3D reconstructions of the target particle, it is necessary to obtain a representative 2D projection of the target particle that captures its structural information with as few artifacts as possible. (See reference...) Figure 3The described process typically includes: identifying 2D grain projections in a 2D photomicrograph image; classifying multiple 2D grain projections into a set of categories; determining a corresponding category representation image representing the 2D grain projections in each category; and selecting representative 2D projections from the category representation images of that set of categories. In some cases, the process for generating the representative 2D projections performs multiple iterations. That is, in each iteration after the first iteration, the set of 2D grain projections in the 2D photomicrograph image is identified using a template based on the representative 2D projections selected by the previous iteration.

[0047] At position 250, the system uses this set of representative 2D projections to reconstruct a 3D image of the target particle. The system can use any suitable algorithm to perform 3D reconstruction from the 2D projections, such as by using Fourier transform and backprojection techniques. In some implementations, the system can use an iterative approach to refine the 3D reconstruction. For example, the system can generate an initial 3D reconstruction of the target particle and perform an iterative process that includes: reprojecting the 3D reconstruction back into 2D space to generate a simulated projection, comparing the simulated projection with representative 2D projections, and adjusting the 3D reconstruction to minimize the difference between the simulation and observation.

[0048] Since generating representative 2D projections can be computationally intensive, especially when processing high-resolution photomicrographs, strategies can be employed to optimize efficiency. In some implementations, the system can perform step 240 to generate an initial representative 2D projection based on a downsampled image (e.g., a reduced-resolution image), and then use this initial representative 2D projection as a guide to obtain a high-resolution representative 2D projection for 3D reconstruction. Downsampling involves reducing the resolution and number of pixels of the original high-resolution photomicrograph. This significantly reduces the computational load of steps 240, such as particle identification, classification, averaging, and selection. By using a smaller image size, the system can quickly identify particle projections, thus speeding up step 240. The initial representative 2D projection obtained from the downsampled image can be used to guide the selection of corresponding particles in the original high-resolution image. Once the high-resolution particles are selected, the system can generate a high-resolution representative 2D projection for 3D reconstruction (e.g., category averaging). This approach improves computational efficiency while maintaining reconstruction quality. The initial stage is accelerated by using a smaller image, while the final 3D reconstruction is performed based on full-resolution data to ensure the highest quality 3D reconstruction.

[0049] At position 260, the system outputs a 3D image of the target particles. In some embodiments, the system may output the 3D reconstruction in a specific file format representing the structural information of the 3D reconstruction and its associated metadata. In some embodiments, the system may output the 3D reconstruction to a display device and use 3D visualization tools that allow users to visualize and explore the reconstruction via the display device and user input devices.

[0050] Figure 3 This is a flowchart illustrating an example of a subprocess 240 for generating a set of representative 2D projections of target particles from a 2D micrograph image. For convenience, subprocess 240 will be described as being executed by a system consisting of one or more computers located in one or more locations. For example, an image processing system appropriately programmed according to this disclosure (e.g., Figure 1 The image processing system 130 can execute process 240.

[0051] At position 310, the system identifies multiple 2D particle projections in a 2D photomicrograph. Each 2D particle projection is a corresponding cropped portion of one of the 2D photomicrographs, depicting a single instance of the 2D projection of a target particle at a certain angle. To identify 2D particle projections, the system can first identify the location of the 2D particle projection in the corresponding 2D photomicrograph (e.g., the coordinates of the center), and then extract the 2D particle projection by cropping a portion of the 2D photomicrograph corresponding to the region of interest (ROI) of the 2D particle projection. The system can use any suitable algorithm to identify the 2D particle projections. For example, the system can use a template-based algorithm. Alternatively or additionally, the system can use a template-free algorithm.

[0052] Template-based algorithms rely on predefined templates or reference images of the target particles. These templates can be generated based on prior knowledge of the particle's appearance and structure, user input, and / or have already been generated earlier in the processing pipeline. The algorithm compares the micrographs with the templates to find regions that closely match the expected particle shape and features. This matching process can be performed using techniques such as cross-correlation, template matching, or machine learning-based methods.

[0053] Template-free algorithms do not rely on predefined templates. Instead, they employ techniques based on inherent features, patterns, and statistical properties within photomicrographs to identify particles. For example, these algorithms can use methods such as local variance analysis, edge detection, or machine learning to identify regions that deviate from background noise. Machine learning methods can involve training models to distinguish between particles and noise.

[0054] At position 320, the system classifies multiple 2D particle projections into a group of categories. The goal of classification is to group particle projections that are at the same (or similar) projection angle. The system can use any appropriate technique to perform particle projection classification, including, for example, principal component analysis (PCA), multivariate statistical analysis, template matching, feature extraction-based techniques and / or supervised or unsupervised machine learning.

[0055] At 330, for each category in the group of categories, the system determines a corresponding category representation image representing the 2D particle projection in the corresponding category. For example, the category representation image could be a category average image, which is obtained by averaging the 2D particle projections in the corresponding category. By performing in-category averaging, the category average image can achieve an improved signal-to-noise ratio (SNR) compared to individual images in the category.

[0056] At 340, the system obtains metadata for each category-represented image and uses this metadata to select a representative 2D projection from the category-represented images of that category. The metadata may include one or more of the following: (i) the average pixel value of the category-represented image, (ii) the maximum pixel value of the category-represented image, (iii) the minimum pixel value of the category-represented image, (iv) the standard deviation of the pixel values ​​of the category-represented image, (v) the signal-to-noise ratio (SNR) of the pixel values ​​of the category-represented image, and / or (vi) a category distribution value characterizing the number of 2D grain projections within the category of the corresponding category-represented image. In this specification, a pixel value refers to the value assigned to a pixel in an image, such as the pixel's intensity value. In some cases, the SNR is calculated as the ratio of the maximum pixel value to the minimum pixel value, i.e., .

[0057] In some implementations, to select a group of representative 2D projections from the categorical representation images, the system can exclude categorical representation images from the selection if one or more parameter values ​​in the metadata of the categorical representation images fall outside a specific range. That is, for each corresponding category, the system determines whether each of one or more parameter values ​​is within a corresponding range (e.g., defined by one or more corresponding thresholds for the parameter value). If each of one or more parameter values ​​is within the corresponding range, the system includes the corresponding categorical representation image in the selection. On the other hand, if at least one parameter value is not within the specific range, the system excludes the corresponding categorical representation image from the selection. For example, if the average pixel value of a categorical representation image exceeds a specific range, or if the SNR of a categorical representation image exceeds a specific range, the system can exclude the corresponding categorical representation image from the selection. In some cases, for each corresponding parameter value, the corresponding range is determined based on a predefined quantile of the parameter values ​​of the group of categorical representation images. For example, for the average pixel value, the specific range may be set to 90% of the average pixel value of the group of categorical representation images. That is, if the average pixel value of a particular categorical representation image is within the highest 10% of all categorical representation images, the particular categorical representation image is considered an outlier and excluded from the selected group of representative 2D projections.

[0058] Figure 4 An example of metadata for a representative 2D grain projection is shown. The metadata includes average pixel value, standard deviation of pixel value, maximum pixel value, minimum pixel value, pixel value SNR, and category distribution value.

[0059] Return to reference Figure 3In some implementations, to select the group of representative 2D projections (at 340) from the class representation images using metadata, the system can use a machine learning model to process the metadata to generate a predictive output for selecting the representative 2D projection. For each corresponding class representation image, the system can generate a corresponding model input for the corresponding class representation image using the corresponding metadata, process the corresponding model input using the machine learning model to generate a corresponding output, and determine whether to include the corresponding class representation image in the selection based on the output. The machine learning model can include any suitable predictive machine learning model, such as a neural network and / or a random forest model. The input to the machine learning model can be a vector including parameters from the metadata, such as the average pixel value, maximum pixel value, minimum pixel value, standard deviation of pixel values, pixel value SNR, and class distribution value of the corresponding class representation image. In some implementations, the output of the machine learning model can be a classification indicating whether the corresponding class representation image should be included in the selection. In some implementations, the output of the machine learning model can be a predicted score predicting the probability that the corresponding class representation image belongs to the selection. In a particular example, the machine learning model is a random forest model comprising multiple decision trees. During training, each tree is trained independently on a random subset of the training data. The training process involves randomly sampling instances with replacement, which introduces diversity among the trees and helps reduce overfitting. During inference, the predictions from the individual trees are combined using a majority voting mechanism. The class that receives the most votes across all trees is the final predicted class.

[0060] In some implementations, the process including steps 310 through 340 may be performed multiple times. In each iteration following the first iteration, the system uses a template based on the set of representative 2D projections selected by the previous iteration to identify a set of 2D grain projections in the 2D photomicrograph image (at 310). That is, after the system performs an iteration at 340, the system determines whether to continue iterating (at 350). If the system determines to continue iterating, the system uses the representative 2D projections as the grain template (at 370) and continues to the next iteration, in which the system uses a template based on the set of representative 2D projections selected by the previous iteration to identify 2D grain projections in the 2D photomicrograph image (at 310). When a stopping condition is met at 350, the system completes updating the representative 2D projections at 360. The stopping condition may include (i) a predefined number of iterations has been performed, (ii) changes to the updated representative 2D projections are within a certain threshold, or (iii) one or more quality parameters (e.g., SNR) of all updated representative 2D projections are within a predefined range. This iterative process can repeatedly refine representative 2D projections to improve their quality. The iterative process can be predefined, for example, within workflow configuration data.

[0061] In some other implementations, the process including steps 320 to 340 can be performed multiple times. In each iteration, the system performs an initial classification (320) on the 2D grain projection, generates a category representation image (330), obtains metadata of the category representation image, and uses the metadata to select a representative 2D projection (340), for example, determining whether to select a representative 2D projection based on whether one or more parameter values ​​in the metadata of the corresponding category representation image are within a range defined by one or more corresponding thresholds of the parameter values. At the end of the iteration, the system determines whether to continue the iteration (at 350). If so, the system continues to the next iteration, in which the system again performs classification (at 320) on the 2D grain projections corresponding to the set of representative 2D projections selected (at 340) in the previous iteration. When the stopping condition is met at 350, the system completes the update of the representative 2D projections at 360. This iterative process can be repeated to refine the representative 2D projections to improve their quality. The iterative process can be predefined, for example, in workflow configuration data.

[0062] In some cases, the system can adjust the threshold over multiple iterations used to select representative 2D projections. For example, at iteration 340, if the SNR of the class representation image is higher than the corresponding SNR threshold of the initial iteration, the system can select that class representation image as one of the selected representative 2D projections. As the system proceeds to the next iteration, it can adjust the SNR threshold used to select representative 2D projections at iteration 340 (e.g., in a predefined increment). That is, in the initial iteration, the system can use a lower SNR threshold to include a wider range of projections. As iterations progress, this threshold can be adjusted incrementally, making the selection criteria more stringent and focusing on projections with progressively higher SNR values. This approach allows for a more granular refinement process, starting with a wider selection and progressively narrowing it down to representative projections with optimal quality. While the above example focuses on using SNR to select representative 2D projections, the system can utilize other metadata parameters for selection. These parameters can include resolution estimates or quality scores predicted by an AI model, or any other relevant metrics that can assess the quality and representativeness of a projection.

[0063] Figure 5This is a block diagram of an example computer system 500 that can be used to perform the operations described above. System 500 includes a processor 510, memory 520, storage device 530, and input / output device 540. Each of components 510, 520, 530, and 540 may be interconnected, for example, using a system bus 550. Processor 510 is capable of processing instructions for execution within system 500. In one embodiment, processor 510 is a single-threaded processor. In another embodiment, processor 510 is a multi-threaded processor. Processor 510 is capable of processing instructions stored in memory 520 or on storage device 530.

[0064] The memory 520 stores information within the system 500. In one embodiment, the memory 520 is a computer-readable medium. In one embodiment, the memory 520 is a volatile memory cell. In another embodiment, the memory 520 is a non-volatile memory cell.

[0065] Storage device 530 provides high-capacity storage for system 500. In one embodiment, storage device 530 is a computer-readable medium. In various other embodiments, storage device 530 may include, for example, a hard disk drive, an optical disk drive, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other high-capacity storage device.

[0066] Input / output device 540 provides input / output operations for system 500. In one embodiment, input / output device 540 may include one or more network interface devices (e.g., Ethernet cards), serial communication devices (e.g., RS-232 ports), and / or wireless interface devices (e.g., 502.11 cards). In another embodiment, input / output device may include a driver device configured to receive data and send output data to other input / output devices (e.g., keyboards, printers, and display device 760). However, other embodiments may also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0067] Despite Figure 5 Example processing systems have been described in this disclosure, but implementations of the subjects and functional operations described herein may be carried out in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed herein and their structural equivalents), or a combination thereof.

[0068] This disclosure uses the term "configuration" when referring to system and computer program components. For a system comprising one or more computers to be configured to perform specific operations or actions, this means that the system has software, firmware, hardware, or combinations thereof installed thereon that causes the system to perform those operations or actions during runtime. For one or more computer programs to be configured to perform specific operations or actions, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions. Embodiments of the subject matter and functional operations described in this disclosure may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this disclosure and their structural equivalents), or combinations thereof. Embodiments of the subject matter described in this disclosure may be implemented as one or more computer programs (i.e., one or more computer program instruction modules encoded on a tangible non-transitory storage medium) for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0069] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0070] Computer programs (which may also be referred to or described as programs, software, software applications, applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and they can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program can be deployed to execute on a single computer, or on multiple computers located in one location or distributed across multiple locations and interconnected via a data communication network.

[0071] In this disclosure, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way or at all, and the data can be stored on storage devices in one or more locations. Therefore, for example, an index database may include multiple collections of data, each of which may be organized and accessed in different ways.

[0072] Similarly, in this disclosure, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed on or run on the same one or more computers.

[0073] The processes and logical flows described in this disclosure can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. These processes and logical flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0074] A computer suitable for executing computer programs can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or on any other type of central processing unit (CPU). Typically, the CPU receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the CPU for executing instructions and one or more memory devices for storing instructions and data. The CPU and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to, or both. However, a computer does not necessarily need to have such a device. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive).

[0075] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0076] To provide interaction with the user, embodiments of the subject matter described in this disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including auditory, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages returned by the user.

[0077] The data processing apparatus used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the computationally intensive portions common in machine learning training or production (i.e., inference) workloads.

[0078] Machine learning models can be implemented and deployed using machine learning frameworks such as PyTorch or TensorFlow.

[0079] Embodiments of the subject matter described in this disclosure can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), front-end components (e.g., a client computer having a graphical user interface, web browser, or application through which a user can interact with embodiments of the subject matter described in this disclosure), or any combination of one or more such back-end components, middleware, or front-end components. Components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0080] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device acting as a client, for example, to display data to a user interacting with that device and to receive user input from that user. Data generated at the user device (e.g., the result of user interaction) can be received from that device at the server.

[0081] While this disclosure contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope of what may be claimed, but rather as a description of features that may be specific to a particular embodiment of the invention. Certain features described in this disclosure, in the context of a single embodiment, may also be implemented in combination in a single embodiment. Conversely, different features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in a certain combination and even initially claimed in this way, in some cases, one or more features from the claimed combination may be removed from that combination, and the claimed combination may involve sub-combinations or variations thereof.

[0082] Similarly, although operations are depicted in a specific order in the accompanying drawings and are also recited in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or requiring the performance of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

Claims

1. A computer-implemented method for determining the three-dimensional (3D) structure of a class of particles in a sample, the method comprising: Obtain one or more two-dimensional (2D) micrographs of the sample; Perform a process for generating a set of representative 2D projections of this type of particle from these 2D micrographs, the process including: Identify multiple 2D particle projections in these 2D micrographs; Classify the multiple 2D particle projections into a single category; For each category in this group of categories, determine the corresponding category representation image that represents the projection of these 2D particles in that corresponding category; Obtain the corresponding metadata for each category representing the image, which includes one or more of the following: the average pixel value, maximum pixel value, minimum pixel value, standard deviation of pixel values, or signal-to-noise ratio (SNR) of pixel values ​​in the image representing that category; and Using the corresponding metadata for each category-represented image, select the group of representative 2D projections from these category-represented images; Using this set of representative 2D projections, 3D images of this type of particle were reconstructed; and Output a 3D image of this type of particle.

2. The method of claim 1, wherein, Determining the category representation image representing the projection of these 2D particles in the corresponding category includes: The average image is obtained by averaging these multiple 2D particle projections in the corresponding category to serve as the image representing that category.

3. The method of claim 1 or claim 2, wherein, The process for generating the group of representative 2D projections is performed in multiple iterations, wherein in each iteration after the first iteration, the group of 2D particle projections in the 2D photomicrograph are identified using a template based on the group of representative 2D projections selected by the previous iteration.

4. The method of any preceding claim, wherein, The corresponding metadata further includes a category distribution value, which characterizes the number of 2D grain projections in that category of the image represented by the corresponding category.

5. The method of any preceding claim, wherein, The selection of representative 2D projections from these category representation images for each category includes, for each corresponding category: Determine one or more parameter values ​​from the corresponding metadata representing the image within the corresponding category; Determine whether each of the one or more parameter values ​​is within a corresponding range defined by one or more corresponding thresholds for that parameter value; In response to determining that each of the one or more parameter values ​​falls within the corresponding range, the image representing the corresponding category is included in the selection; and In response to determining that at least one of these parameter values ​​is not within the corresponding range, the corresponding category of the image is excluded from the selection.

6. The method of claim 5, wherein, The process for generating the representative 2D projections is performed in multiple iterations, wherein in each iteration after the first iteration, the classification of the multiple 2D particle projections is performed on a subset of the 2D particle projections corresponding to the representative 2D projections selected in the previous iteration.

7. The method of claim 6, wherein, At least one of the one or more corresponding thresholds is adjusted in the multiple iterations.

8. The method of any one of claims 5 to 7, wherein, For each corresponding parameter value, the corresponding range is determined based on the corresponding predefined quantile of that parameter value.

9. The method of any one of claims 5 to 8, wherein, The one or more parameter values ​​used to select the representative 2D projections include the average pixel value and the pixel value signal-to-noise ratio.

10. The method of claim 9, wherein, The signal-to-noise ratio of this pixel value is calculated as the ratio of the maximum pixel value to the minimum pixel value.

11. The method of any preceding claim, wherein, Using the corresponding metadata for each category-represented image, the representative 2D projection of the group of category-represented images is selected from these category-represented images, including, for each corresponding category-represented image: Use the corresponding metadata to generate the corresponding model input for the image representing the corresponding category; The corresponding model input is processed using a machine learning model to generate the corresponding output; and Based on this output, determine whether to include the corresponding category representation image in the selection.

12. The method of claim 11, wherein, The inputs for generating the corresponding model include: Generate a corresponding vector using the average pixel value, the maximum pixel value, the minimum pixel value, the standard deviation of the pixel value or the SNR of the pixel value, and the category distribution value; and The corresponding model input is generated from the corresponding vector.

13. The method of any preceding claim, wherein, These 2D micrographs include electron microscope (EM) images of the sample.

14. The method of claim 13, wherein, These 2D micrographs include a set of EM images of the sample taken from different angles.

15. The method of any of the preceding claims, further comprising: Before performing the process used to generate this set of representative 2D projections, these 2D photomicrographs are processed to correct motion artifacts.

16. The method of any of the preceding claims, further comprising: Before performing the process for generating the set of representative 2D projections, the contrast transfer function (CTF) of these 2D photomicrographs is estimated and the 2D photomicrographs are processed to reduce blur caused by the CTF.

17. The method of any of the preceding claims, further comprising: Receive configuration data that defines the workflow for performing the process of generating the representative 2D projections of the set; And reconstruct the 3D image using the set of representative 2D projections; and perform these operations based on the workflow defined in the configuration data.

18. The method of claim 17, wherein, This configuration data defines one or more loops or repetitive steps in these operations.

19. The method of claim 18, wherein, This configuration data defines one or more algorithms or options used in these operations.

20. The method of claim 19, wherein, The configuration data includes one or more labels that mark these repetitive steps.

21. A system comprising: One or more computers; as well as One or more storage devices that store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of the corresponding method as described in any one of claims 1 to 20.

22. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding method as claimed in any one of claims 1 to 20.