Training method and device of cryo-em real space decoder and electronic equipment

Abstract

The application provides a training method and device of a cryo-EM real space decoder and electronic equipment. The method comprises: obtaining input projection images and label information of cryo-EM; inputting projection parameters of the input projection images into the decoder to output predicted projection images; performing mask processing on a machine learning representation form of a three-dimensional reconstructed target structure, the predicted projection images and the input projection images through a mask; calculating a loss value based on the input projection images and the predicted projection images after the mask processing, and adjusting parameters of the decoder based on the loss value; and until a preset training condition is met, a trained decoder is obtained. The method is suitable for a machine learning scene, and can make the cryo-EM three-dimensional reconstruction based on machine learning meet actual demands, focus on key local information of the three-dimensional reconstructed target structure, effectively separate foreground and background, and thus improve reconstruction quality, and good results can be achieved when the number of images is small.

Description

Technical Field

[0001] This invention relates to the field of cryo-electron microscopy, and in particular to a training method, apparatus, and electronic device for a cryo-electron microscopy real-space decoder. Background Technology

[0002] Single-particle cryo-electron microscopy (SPE) is an important experimental technique in structural biology. This technique uses two-dimensional images of biomolecules obtained from electron microscopy to resolve their three-dimensional structures, thereby helping biologists understand the functional mechanisms of biomolecules and aiding in new drug design. Three-dimensional reconstruction is a core step in SPE, enabling the conversion from two-dimensional electron microscope images to three-dimensional molecular density maps.

[0003] Traditional cryo-electron microscopy (cryo-EM) 3D reconstruction algorithms can be divided into two main categories: algebraic reconstruction algorithms and Fourier transform algorithms. Fourier transform-based algorithms for 3D reconstruction in frequency space are currently widely used, but they suffer from the following problems: These algorithms require interpolation calculations in the frequency space, and it's difficult to achieve both speed and accuracy in this interpolation method. Commonly used fast interpolation algorithms can lead to imperfections due to accuracy loss and resolution loss in the reconstructed 3D density map. This deficiency is particularly pronounced with small datasets, as they lack more information in the frequency space. Furthermore, the frequency domain representation only preserves a description of the global structure and is insensitive to the irregularities of the local structure in real space, making it difficult to optimize for specific spatial regions.

[0004] While traditional algebraic algorithms can avoid these two problems, they are inherently computationally inefficient and cannot be directly applied to machine learning frameworks. Consequently, their computational efficiency is far inferior to machine learning-accelerated 3D reconstruction algorithms, and they are difficult to integrate with other machine learning algorithms, such as machine learning-based 3D classification of electron microscopy images. Therefore, these methods hinder the development of efficient real-space 3D reconstruction algorithms based on machine learning. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a training method, apparatus and electronic device for a cryo-electron microscopy real-space decoder, so that the advantages of cryo-electron microscopy real-space 3D reconstruction can be fully utilized within the framework of machine learning.

[0006] In a first aspect, embodiments of the present invention provide a training method for a cryo-electron microscope real-space decoder. The method includes: acquiring an input projection image of a cryo-electron microscope and label information of the input projection image from a sample set; wherein the label information of the input projection image includes: projection parameters and a mask of the input projection image, the projection parameters including projection angle and image translation; inputting the projection parameters of the input projection image into the decoder, performing spatial transformation on the projection field of the machine learning representation of the 3D reconstructed target structure inside the decoder, and outputting a predicted projection image; wherein the machine learning representation includes an explicit representation and an implicit representation; performing masking processing on the machine learning representation of the 3D reconstructed target structure, the predicted projection image, and the input projection image through a mask; calculating a loss value based on the masked input projection image and the predicted projection image, and adjusting the parameters of the decoder based on the loss value; repeatedly executing the step of acquiring the input projection image of a cryo-electron microscope and label information of the input projection image from the sample set until a preset training condition is met, thereby obtaining a trained decoder.

[0007] In optional embodiments of this application, the above-mentioned mask includes a 3D local mask, a 2D local mask, and / or a planar foreground / background mask.

[0008] In optional embodiments of this application, the 3D local mask includes an inscribed spherical mask or a weighted mask, and the 2D local mask is determined based on the 3D local mask.

[0009] In an optional embodiment of this application, a 2D local mask is obtained by spatial transformation and projection calculation based on projection parameters on the 3D local mask.

[0010] In an optional embodiment of this application, the planar foreground / background mask is used to distinguish the foreground region and background region of the projected image.

[0011] In optional embodiments of this application, the explicit representation of the three-dimensional reconstruction target structure includes: encoding the Coulomb potential value of the three-dimensional structure through the grid points of the three-dimensional grid matrix; the implicit representation of the three-dimensional reconstruction target structure includes: using a neural network to represent the function that maps spatial coordinates to the Coulomb potential value of the three-dimensional structure.

[0012] In an optional embodiment of this application, after the step of inputting the projection parameters of the input projection image into the decoder, the method further includes: performing spatial transformation and subsequent processing on the projection field of the machine learning representation of the three-dimensional reconstructed target structure inside the decoder, and outputting a predicted projection image.

[0013] In an optional embodiment of this application, the step of calculating the loss value based on the masked input projection image and the predicted projection image includes: performing masking processing on the input projection image based on a 2D local mask and a planar foreground and background mask to obtain a masked input projection image; performing masking processing on the predicted projection image based on the planar foreground and background mask to obtain a masked predicted projection image; and determining the loss value of a specified region of the input projection image based on the masked input projection image and the predicted projection image.

[0014] In an optional embodiment of this application, the input projection image after masking includes a designated area and other areas; after obtaining the input projection image after masking, the method further includes: suppressing noise in other areas by regularization.

[0015] In an optional embodiment of this application, the above method further includes: inputting a first electron microscope image from a cryo-electron microscope into a projection parameter prediction encoder and outputting spatial transformation parameters of the first electron microscope image; wherein the spatial transformation parameters characterize the projection parameters of the first electron microscope image; inputting the spatial transformation parameters into a cryo-electron microscope real space decoder and performing training iterations; wherein the loss value generated by the training iterations is back-transmitted to the encoder and decoder through a machine learning computation graph, and the parameters of the encoder and decoder are updated based on the loss value generated by the training iterations.

[0016] Secondly, embodiments of the present invention also provide a training apparatus for a cryo-electron microscope real-space decoder. The apparatus includes: an input projection image acquisition module, used to acquire input projection images of the cryo-electron microscope and label information of the input projection images from a sample set; wherein, the label information of the input projection images includes: projection parameters and a mask of the input projection images, and the projection parameters include projection angle and image translation; a prediction projection image output module, used to input the projection parameters of the input projection images into the decoder, perform spatial transformation on the projection field of the machine learning representation of the three-dimensional reconstructed target structure inside the decoder, and output a prediction projection image; wherein, the machine learning representation includes explicit representation and implicit representation; a mask processing module, used to perform mask processing on the machine learning representation of the three-dimensional reconstructed target structure, the prediction projection image, and the input projection image through a mask; a loss value calculation module, used to calculate a loss value based on the masked input projection image and the prediction projection image, and adjust the parameters of the decoder based on the loss value; and a decoder training completion module, used to repeatedly execute the step of acquiring input projection images of the cryo-electron microscope and label information of the input projection images from the sample set until a preset training condition is met, thereby obtaining a trained decoder.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, characterized in that it includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, and the processor executing the computer-executable instructions to implement the above-described training method for the cryo-electron microscope real-space decoder.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions, which, when called and executed by a processor, cause the processor to implement the above-described training method for the cryo-electron microscope real-space decoder.

[0019] The embodiments of the present invention bring the following beneficial effects:

[0020] The present invention provides a training method, apparatus, and electronic device for a cryo-electron microscope real-space decoder, which is suitable for machine learning scenarios. It makes machine learning-based electron microscope real-space 3D reconstruction better meet practical needs, can focus on key local information of the target structure in 3D reconstruction, effectively separate the foreground and background, thereby improving the reconstruction quality, avoiding the drawbacks of frequency domain 3D reconstruction interpolation algorithms, and can achieve relatively good results even when there are few images.

[0021] Other features and advantages of this disclosure will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0022] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a training method for a cryo-electron microscopy real-space decoder provided in an embodiment of the present invention;

[0025] Figure 2 A flowchart illustrating another training method for a cryo-electron microscopy real-space decoder provided in an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a decoder algorithm provided in an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of an algorithm with an encoder and a decoder provided in an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of the structure of a training device for a cryo-electron microscope real-space decoder provided in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Single-particle cryo-electron microscopy is an important experimental technique in structural biology. This technique uses two-dimensional images of biological macromolecules obtained from electron microscopy to resolve their three-dimensional structures, thereby helping biologists understand the functional mechanisms of biological macromolecules and assisting in new drug design.

[0032] Three-dimensional reconstruction is a core step in single-particle cryo-electron microscopy, enabling the conversion from a two-dimensional electron microscope image to a three-dimensional molecular density map. This embodiment focuses on achieving the reconstruction from a two-dimensional projection image to a three-dimensional object under given key parameters for three-dimensional electron microscope reconstruction (spatial angle / image translation).

[0033] Currently, traditional cryo-electron microscopy (cryo-EM) 3D reconstruction algorithms can be divided into two main categories. The first is algebraic reconstruction algorithms (such as SIRT and ART), which directly utilize the physical imaging principles of electron microscope images to estimate the 3D volume of a corresponding set of electron microscope projection images. The second category is Fourier transform algorithms (such as the methods used in Relion and cryoSPARC electron microscope software). These algorithms are based on the Fourier slice theorem, transforming the projection problem into an interpolation problem in frequency space.

[0034] Machine learning-based 3D reconstruction methods (such as cryoDRGN) are generally designed based on the Fourier slice theorem, using neural networks to fit a function that describes the 3D representation of the target object in frequency space, thereby achieving 3D reconstruction.

[0035] However, algorithms based on Fourier transforms for 3D reconstruction in frequency space are currently widely used, but they suffer from the following problems: These algorithms require interpolation calculations in frequency space, and it's difficult to achieve both speed and accuracy in this interpolation process. Commonly used fast interpolation algorithms can lead to imperfections due to accuracy loss and resolution loss in the reconstructed 3D density map. This deficiency is particularly pronounced on small datasets, as they contain more missing information in frequency space. Furthermore, the frequency domain representation only preserves a description of the global structure and is insensitive to the irregularities of the local structure in real space, making it difficult to optimize for specific spatial regions.

[0036] While traditional algebraic algorithms can avoid these two problems, they are inherently computationally inefficient and cannot be directly applied to machine learning frameworks. Consequently, their computational efficiency is far inferior to machine learning-accelerated 3D reconstruction algorithms, and they are difficult to integrate with other machine learning algorithms, such as machine learning-based 3D classification of electron microscopy images. Therefore, these methods hinder the development of efficient real-space 3D reconstruction algorithms based on machine learning.

[0037] Based on this, the present invention provides a training method, apparatus and electronic device for a cryo-electron microscopy real-space decoder, specifically relating to a masked three-dimensional reconstruction decoder for cryo-electron microscopy real-space, suitable for machine learning.

[0038] Most existing 3D reconstructions are performed in frequency space. Limited by the accuracy of frequency space signal interpolation algorithms, the 3D reconstruction results in frequency space are prone to distortion when reconstructing high-frequency real-space signals, leading to inaccurate results, especially on small datasets. Furthermore, frequency space reconstruction is insensitive to irregularities in local spatial structure, making it difficult to achieve local optimization of the target volume. Traditional real-space 3D reconstruction algorithms cannot be directly applied to machine learning frameworks without modification. The method provided in this embodiment avoids the shortcomings of frequency space 3D reconstruction and is also applicable to machine learning scenarios.

[0039] To facilitate understanding of this embodiment, a training method for a cryo-electron microscopy real-space decoder disclosed in this embodiment will first be described in detail.

[0040] Example 1:

[0041] This invention provides a training method for a cryo-electron microscopy real-space decoder, see [link to documentation]. Figure 1 The flowchart shown illustrates a training method for a cryo-electron microscopy real-space decoder, which includes the following steps:

[0042] Step S102: Obtain the input projection image and label information of the cryo-electron microscope from the sample set; wherein, the label information of the input projection image includes: the projection parameters and mask of the input projection image, and the projection parameters include the projection angle and image translation.

[0043] The decoder in this embodiment is a network capable of predicting the projection image from the electron microscope projection angle and projection translation. During decoder training, the input projection image from the cryo-electron microscope and its label information are used as samples. The label information includes: the projection angle of the input projection image, image translation, and the mask. The mask in this embodiment is not necessarily a single mask; it can include multiple masks, such as 3D local masks, 2D local masks, and planar foreground / background masks.

[0044] Step S104: Input the projection parameters of the input projection image into the decoder, perform spatial transformation on the projection field of the machine learning representation of the 3D reconstructed target structure inside the decoder, and output the predicted projection image.

[0045] Among them, the machine learning representation includes explicit representation and implicit representation. The decoder can perform spatial transformation on the explicit representation, implicit representation or projection field of the 3D reconstructed target structure (3DR) according to the input projection parameters, and the decoder can output the predicted projection image.

[0046] For example, spatial transformations can be performed on the projection fields of explicit and implicit representations, or spatial transformations and subsequent processing can be performed directly on the explicit representation to output a predicted projection image.

[0047] Step S106: Mask the machine learning representation of the 3D reconstructed target structure, the predicted projection image, and the input projection image using a mask.

[0048] This embodiment can perform masking on the machine learning representation of the 3D reconstructed target structure, the predicted projection image, and the input projection image. Through masking, it can be ensured that the predicted projection image and the input projection image use the same region for subsequent calculations.

[0049] Step S108: Calculate the loss value based on the input projection image and the predicted projection image after masking, and adjust the parameters of the decoder based on the loss value.

[0050] Based on a pre-set loss function, the loss values ​​of the input projection image and the predicted projection image after masking can be calculated. The parameters of the decoder can be adjusted according to the loss values, thereby completing one iteration of the decoder.

[0051] Step S110 involves repeatedly performing the step of obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set until the preset training conditions are met, and a trained decoder is obtained.

[0052] In this embodiment, the total number of iterations (e.g., 3000 times) can be preset as a training condition. When the number of iterations reaches this total number, it can be considered that the preset training condition is met, and the decoder is obtained after training.

[0053] The present invention provides a training method for a cryo-electron microscope real-space decoder, which is suitable for machine learning scenarios. It makes machine learning-based electron microscope real-space 3D reconstruction better meet practical needs. It can focus on key local information of the target structure in 3D reconstruction, effectively separate the foreground and background, thereby improving the reconstruction quality. It avoids the drawbacks of frequency domain 3D reconstruction interpolation algorithms and can achieve good results even when there are few images.

[0054] Example 2:

[0055] This embodiment provides another training method for a cryo-electron microscopy real-space decoder, which is implemented based on the above embodiment, such as... Figure 2 The flowchart shown illustrates another method for training a cryo-electron microscopy real-space decoder. The training method for this cryo-electron microscopy real-space decoder in this embodiment includes the following steps:

[0056] Step S202: Obtain the input projection image and label information of the cryo-electron microscope from the sample set; wherein, the label information of the input projection image includes: the projection parameters and mask of the input projection image, and the projection parameters include the projection angle and image translation.

[0057] For easier understanding, please refer to Figure 3 The diagram illustrates a decoder algorithm, which may include a training process and an inference process. During training, electron microscope images and their label information are input into the network for training. The training process terminates when the training loss no longer decreases or a certain number of training iterations are reached.

[0058] The input projection images and their label information used in the training process include cryo-electron microscopy projection images obtained in the experiment, and the label information includes: the projection angle of the input projection image, image translation, and mask, etc.

[0059] Step S204: Input the projection parameters of the input projection image into the decoder, perform spatial transformation on the projection field of the machine learning representation of the 3D reconstructed target structure inside the decoder, and output the predicted projection image.

[0060] Machine learning representations include explicit and implicit representations. In this embodiment, the projection field of the machine learning representation of the 3D reconstructed target structure inside the decoder can also be spatially transformed and subsequently processed to output a predicted projection image.

[0061] The explicit representation of the three-dimensional reconstruction target structure includes: encoding the Coulomb potential value of the three-dimensional structure through the grid points of the three-dimensional grid matrix; the implicit representation of the three-dimensional reconstruction target structure includes: using a neural network to represent the function that maps spatial coordinates to the Coulomb potential value of the three-dimensional structure.

[0062] One implementation of explicit 3DR is a three-dimensional lattice matrix, where each lattice point can be encoded with a numerical value or a parameterized function (such as a spherical harmonic function) to determine the Coulomb potential of the three-dimensional structure. Another implementation of implicit 3DR uses a neural network to represent a function that can calculate the Coulomb potential of the three-dimensional structure for each coordinate value in the corresponding space.

[0063] like Figure 3 As shown, during the training process, the network needs to input a set of projection parameters, project the 3DR image according to the projection angle, output the projection image at that angle, and calculate the loss function with the input experimental electron microscope image.

[0064] For explicit 3DR 2D planar projection calculations, the 3D grid matrix can be transformed to a new spatial orientation and position based on the input spatial transformation parameters (including rotation and translation information). Then, the voxel information of the original 3D grid matrix is ​​used to perform an interpolation algorithm to calculate the voxel information of the new 3D grid matrix. Finally, integration or mean calculation is performed along the projection direction.

[0065] Furthermore, the aforementioned explicit 3DR can also be replaced by an implicit representation based on neural networks, such as: position-encoding neural networks, networks based on sine / cosine activation functions like SIREN, and networks based on generative techniques such as GAN / VAE / diffusion models.

[0066] When acquiring the projection, this embodiment can use the method of rotating the three-dimensional grid matrix to obtain the projection, or it can use other methods to obtain the projection, such as: without rotating the three-dimensional grid, directly projecting at the corresponding rotation angle and translation position; directly predicting the Coulomb potential value of the three-dimensional reconstructed target object in the projection field of view with the corresponding projection parameters through a neural network and projecting it; using the above methods, processing the 3DR with foreground / background masks, local masks, etc., before projecting.

[0067] Electron microscope projection parameters can be predicted using other specialized software; predicted using numerical search algorithms combined with 3D reconstruction methods; predicted using neural networks suitable for image processing; predicted using neural networks during training, while simultaneously optimizing the prediction network and the 3D reconstruction network; or predicted by adding correction terms using any of the above methods.

[0068] Step S206: Mask the machine learning representation of the 3D reconstructed target structure, the predicted projection image, and the input projection image using a mask.

[0069] Specifically, the mask in this embodiment includes a 3D local mask, a 2D local mask, and / or a planar foreground / background mask. The 3D local mask includes an inscribed spherical mask or a weighted mask. The 2D local mask is determined based on the 3D local mask, for example, by performing spatial transformation and projection calculations on the 3D local mask based on projection parameters to obtain the 2D local mask.

[0070] There are many design schemes for 3D local masks, such as the inscribed spherical mask, which sets the mask value at the corners of the 3DR to zero, while setting the mask value inside the 3DR inscribed sphere to 1. 3D local masks can also be set as weighted masks, that is, different mask weight values ​​(between 0 and 1) are set for different regions of the 3DR.

[0071] The design of 2D local masks is based on 3D local masks. One possible implementation is to obtain the 3D local mask by performing a spatial transformation based on the input projection parameters and then performing projection calculations.

[0072] like Figure 3 As shown, explicit 3DR can be constructed from a three-dimensional grid matrix, while implicit 3DR can be constructed from a neural network. Furthermore, 3DR requires a three-dimensional mask preprocessing step before rotation / translation transformations to avoid spurious signals appearing at the corners of the eight cubes during 3D reconstruction. The three-dimensional mask (i.e., the 3D local mask) used in this embodiment can be the inscribed sphere of the three-dimensional grid matrix; that is, voxel signals within the inscribed sphere are preserved, while voxel signals outside the inscribed sphere are set to zero.

[0073] The decoder provided in this embodiment operates entirely in real space. The 3D mask here can be designed as a weighted 3D foreground mask according to user needs. For example, if the target object for 3D reconstruction contains three spatial regions A, B, and C, and the user is most interested in region A, slightly interested in region B, and not interested in region C, then the 3D mask can assign the highest weight to region A, a lower weight to region B, and a very low weight to region C, or even set the weight to zero.

[0074] A 3D mask can be converted into a 2D mask (i.e., a 2D local mask). For example, a 2D mask can be obtained by performing the same spatial transformation and projection on a 3D mask as on a 3DR mask. This 2D mask can be used to preprocess the input electron microscope image during subsequent loss function calculations to ensure that the predicted image and the input image use the same region for related calculations.

[0075] Step S208: Calculate the loss value based on the input projection image and the predicted projection image after masking, and adjust the parameters of the decoder based on the loss value.

[0076] Specifically, in this embodiment, the input projection image can be masked based on a 2D local mask and a planar foreground and background mask to obtain a masked input projection image; the predicted projection image can be masked based on a planar foreground and background mask to obtain a masked predicted projection image; and the loss value of a specified region of the input projection image can be determined based on the masked input projection image and the predicted projection image.

[0077] In addition, this embodiment can also suppress noise in other regions through regularization.

[0078] like Figure 3 As shown, the purpose of designing a planar foreground / background mask is to distinguish the foreground from the background. Here, the foreground refers to the area in the projection image corresponding to the 3D reconstructed target structure, while the background refers to other areas within the projection image.

[0079] Two-dimensional foreground / background masks can be used to improve reconstruction quality during training. Specifically, in the step of calculating the loss function, two-dimensional foreground / background masks can be applied to the input and predicted projection maps. The model only calculates the loss function for the global or local foreground, while regularization methods are used to suppress noise in the background.

[0080] Foreground / background masks can be designed manually, calculated using specialized software, or predicted directly from projection results. Foreground masking methods can reduce noise while preserving the signal, thereby improving model performance.

[0081] For global foreground masks, this embodiment designs an automated two-dimensional foreground / background mask generation scheme. In this scheme, the input image can be smoothed (e.g., using a Gaussian filter). If the pixel values ​​of the processed input projection image exceed a certain threshold, these areas are designated as foreground, and other areas are designated as background. Then, a binary or weighted foreground / background mask is created based on these areas. The transition region between the foreground and background can be gradually processed using a smoothing function (e.g., the sigmoid function).

[0082] To design a weighted foreground mask, the weights can be calculated based on smoothed pixel values ​​(e.g., through normalization or by inputting these pixel values ​​into a weight generation function to obtain the mask weights). Local foreground masks can be generated using the aforementioned method of converting 3D to 2D region masks. It is worth emphasizing that the same projection image can undergo multiple masking processes.

[0083] When calculating the loss function, L2 loss was used to compare the predicted and input projected images, while L1 loss was used for the reconstructed 3D volume. The final loss function includes both parts. Additionally, a pre-factor beta can be added to the L1 loss for the 3D volume to optimize and reduce the negative impact of noise in the input image on the 3D reconstruction. Preferably, setting beta to 0.05 yields the best results for 3D reconstruction.

[0084] Furthermore, the loss function used in this embodiment can be replaced by other similar functions, including: smooth L1 loss, Huber loss, Hinge loss, cross-entropy loss, etc.; the regularization term can be replaced by other regularization loss functions, including: L1 loss, L2 loss, L1+L2 loss, TV loss, etc.

[0085] Step S210 involves repeatedly performing the step of obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set until the preset training conditions are met, and a trained decoder is obtained.

[0086] This embodiment can use the AdamW optimizer for machine learning iterations. Preferably, setting the weight decay parameter of the AdamW optimizer to 1.0E-6 yields the best reconstruction results.

[0087] Step S212: Generate a 3D reconstruction structure that matches the input projection map dataset.

[0088] During inference, the encoder network can directly output a 3D lattice matrix representing the Coulomb potential energy of the 3D reconstructed target structure as the reconstruction result. The 3D reconstruction algorithm in this embodiment can run on low-end graphics cards and obtain relatively good reconstruction results. Preferably, setting the batch size to 8 achieves a fairly good 3D reconstruction. Setting the learning rate to 0.1 yields the best results.

[0089] Furthermore, in this embodiment, the first electron microscope image from the cryo-electron microscope can be input into the projection parameter prediction encoder to output the spatial transformation parameters of the first electron microscope image; wherein, the spatial transformation parameters represent the projection parameters of the first electron microscope image; the spatial transformation parameters are input into the cryo-electron microscope real decoder and training iteration is performed; wherein, the loss value generated by the training iteration is back-transmitted to the encoder through the machine learning computation graph, and the parameters of the encoder and decoder are updated based on the loss value generated by the training iteration.

[0090] The spatial transformation parameters include various representations of spatial transformations, such as spatial transformation matrices, quaternions, and Rodrigues space rotation vectors, which can be found in [reference needed]. Figure 4 The diagram shows a schematic of an algorithm with an encoder and a decoder.

[0091] Specifically, the electron microscope sample image from the cryo-electron microscope can be input into the initial encoder, which outputs spatial transformation parameters. The spatial transformation parameters of the electron microscope sample image are used as the projection angle of the three-dimensional reconstructed target structure to train the initial encoder until the preset training conditions are met, thus obtaining the trained projection parameter prediction encoder.

[0092] Projection parameters can be predicted by neural networks. For example, the first electron microscope image can be input into the angle prediction network, and the output can be a spatial rotation and translation transformation parameter or an equivalent representation. This transformation parameter can be used as the spatial transformation parameter of 3DR and added to the training process to achieve joint training of the 3D reconstruction network and the projection parameter prediction network.

[0093] In summary, the method provided in this embodiment enables real-space representation during the reconstruction process; a real-space projection algorithm is used in the reconstruction process; and a real-space mask can be applied during the reconstruction process. This approach achieves real-space differentiable electron microscope 3D reconstruction within a machine learning framework, avoiding the various drawbacks of frequency-space 3D reconstruction; and a real-space mask suitable for machine learning is designed to improve the overall quality of the 3D reconstruction and optimize key local features.

[0094] Example 3:

[0095] Corresponding to the above method embodiments, this invention provides a training device for a cryo-electron microscopy real-space decoder, see [link to relevant documentation]. Figure 5 The diagram shows a structural schematic of a training device for a cryo-electron microscopy real-space decoder. The training device includes:

[0096] The input projection image acquisition module 51 is used to acquire the input projection image of cryo-electron microscopy and the label information of the input projection image from the sample set; wherein, the label information of the input projection image includes: the projection parameters and mask of the input projection image, and the projection parameters include the projection angle and image translation;

[0097] The predicted projection image output module 52 is used to input the projection parameters of the input projection image into the decoder, perform spatial transformation on the projection field of the machine learning representation of the three-dimensional reconstructed target structure inside the decoder, and output the predicted projection image; wherein, the machine learning representation includes explicit representation and implicit representation.

[0098] The mask processing module 53 is used to perform mask processing on the machine learning representation of the 3D reconstructed target structure, the predicted projection image and the input projection image through a mask.

[0099] The loss value calculation module 54 is used to calculate the loss value based on the input projection image and the predicted projection image after masking, and to adjust the parameters of the decoder based on the loss value.

[0100] The decoder training completion module 55 is used to repeatedly execute the steps of obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set until the preset training conditions are met, and the trained decoder is obtained.

[0101] The present invention provides a training device for a cryo-electron microscope real-space decoder, which is suitable for machine learning scenarios. It makes machine learning-based electron microscope real-space 3D reconstruction better meet practical needs. It can focus on key local information of the target structure in 3D reconstruction, effectively separate the foreground and background, thereby improving the reconstruction quality. It avoids the drawbacks of frequency domain 3D reconstruction interpolation algorithms and can achieve good results even when there are few images.

[0102] The aforementioned masks include 3D local masks, 2D local masks, and / or planar foreground / background masks.

[0103] The aforementioned 3D local mask includes an inscribed spherical mask or a weighted mask, and the 2D local mask is determined based on the 3D local mask. The 2D local mask is obtained by performing spatial transformation and projection calculations on the 3D local mask based on projection parameters.

[0104] The aforementioned planar foreground and background mask is used to distinguish the foreground and background areas of the projected image.

[0105] The explicit representation of the aforementioned three-dimensional reconstruction target structure includes: encoding the Coulomb potential value of the three-dimensional structure through the grid points of the three-dimensional grid matrix; the implicit representation of the three-dimensional reconstruction target structure includes: using a neural network to represent the function that maps spatial coordinates to the Coulomb potential value of the three-dimensional structure.

[0106] The aforementioned predicted projection image output module is also used to perform spatial transformation and subsequent processing on the projection field of the machine learning representation of the three-dimensional reconstructed target structure inside the decoder, and output the predicted projection image.

[0107] The aforementioned loss value calculation module is used to perform masking processing on the input projection image based on a 2D local mask and a planar foreground and background mask to obtain a masked input projection image; to perform masking processing on the predicted projection image based on a planar foreground and background mask to obtain a masked predicted projection image; and to determine the loss value of a specified region of the input projection image based on the masked input projection image and the predicted projection image.

[0108] The input projection image after the above masking process includes a specified area and other areas; the above device also includes a noise suppression module, which is used to suppress noise in other areas through regularization.

[0109] The aforementioned apparatus further includes: a projection parameter prediction encoder processing module, used to input the first electron microscope image of the cryo-electron microscope into the projection parameter prediction encoder and output the spatial transformation parameters of the first electron microscope image; wherein, the spatial transformation parameters characterize the projection parameters of the first electron microscope image; the spatial transformation parameters are input into the cryo-electron microscope real space decoder and training iteration is performed; wherein, the loss value generated by the training iteration is back-transmitted to the encoder and decoder through a machine learning computation graph, and the parameters of the encoder and decoder are updated based on the loss value generated by the training iteration.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the training device for the cryo-electron microscope real-space decoder described above can be referred to the corresponding process in the aforementioned embodiments of the training method for the cryo-electron microscope real-space decoder, and will not be repeated here.

[0111] Example 4:

[0112] This invention also provides an electronic device for running the training method of the cryo-electron microscopy real-space decoder described above; see also Figure 6 The diagram shows the structure of an electronic device, which includes a memory 100 and a processor 101. The memory 100 is used to store one or more computer instructions, which are executed by the processor 101 to implement the training method of the cryo-electron microscope real-space decoder described above.

[0113] Furthermore, Figure 6 The electronic device shown also includes a bus 102 and a communication interface 103, with the processor 101, the communication interface 103 and the memory 100 connected via the bus 102.

[0114] The memory 100 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 102 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0115] Processor 101 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 101 or by instructions in software form. Processor 101 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 100, and processor 101 reads information from memory 100 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0116] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the training method for the cryo-electron microscope real-space decoder described above. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0117] The computer program product of the training method, apparatus and electronic device for cryo-electron microscope real-space decoder provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and / or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0119] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0120] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0122] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A training method for a cryo-electron microscopy real-space decoder, characterized in that, include: The input projection image of the cryo-electron microscope and the label information of the input projection image are obtained from the sample set; wherein, the label information of the input projection image includes: the projection parameters and mask of the input projection image, and the projection parameters include the projection angle and image translation; The projection parameters of the input projection image are input into the decoder, and a spatial transformation is performed on the projection field of the machine learning representation of the 3D reconstructed target structure within the decoder, outputting a predicted projection image. The machine learning representation includes explicit and implicit representations. The explicit representation of the 3D reconstructed target structure includes encoding the Coulomb potential value of the 3D structure through the grid points of a 3D grid matrix. The implicit representation of the 3D reconstructed target structure includes using a neural network to represent the function that maps spatial coordinates to the Coulomb potential value of the 3D structure. The machine learning representation of the 3D reconstructed target structure, the predicted projection image, and the input projection image are masked using the mask. The loss value is calculated based on the input projection image and the predicted projection image after masking, and the parameters of the decoder are adjusted based on the loss value; The process of repeatedly obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set is repeated until the preset training conditions are met, resulting in a trained decoder.

2. The method according to claim 1, characterized in that, The mask includes a 3D local mask, a 2D local mask, and / or a planar foreground / background mask.

3. The method according to claim 2, characterized in that, The 3D local mask includes an inscribed spherical mask or a weighted mask, and the 2D local mask is determined based on the 3D local mask.

4. The method according to claim 2, characterized in that, The 3D local mask is spatially transformed and projected based on the projection parameters to obtain the 2D local mask.

5. The method according to claim 2, characterized in that, The planar foreground and background mask is used to distinguish the foreground and background areas of the projected image.

6. The method according to claim 1, characterized in that, After the step of inputting the projection parameters of the input projected image into the decoder, the method further includes: The projected field of view of the machine learning representation of the 3D reconstructed target structure inside the decoder is spatially transformed and subsequently processed to output the predicted projected image.

7. The method according to claim 2, characterized in that, The step of calculating the loss value based on the input projection image and the predicted projection image after masking includes: The input projection image is masked based on the 2D local mask and the planar foreground and background mask to obtain the masked input projection image; The predicted projection image is masked based on the planar foreground and background mask to obtain the masked predicted projection image. The loss value of a specified region of the input projection image is determined based on the input projection image after masking and the predicted projection image.

8. The method according to claim 7, characterized in that, The input projection image after masking includes a specified region and other regions; After obtaining the masked input projection image, the method further includes: Noise in the other regions is suppressed by regularization.

9. The method according to claim 1, characterized in that, The method further includes: The first electron microscope image from the cryo-electron microscope is input into the projection parameter predictive encoder, and the spatial transformation parameters of the first electron microscope image are output; wherein, the spatial transformation parameters characterize the projection parameters of the first electron microscope image shown. The spatial transformation parameters are input into the cryo-electron microscope real-space decoder, and training iterations are performed. The loss value generated during training iterations is fed back to the encoder and decoder through a machine learning computation graph, and the parameters of the encoder and decoder are updated based on the loss value generated during training iterations.

10. A training device for a cryo-electron microscopy real-space decoder, characterized in that, The device includes: An input projection image acquisition module is used to acquire input projection images of cryo-electron microscopy and label information of the input projection images from a sample set; wherein, the label information of the input projection images includes: projection parameters and a mask of the input projection images, and the projection parameters include projection angle and image translation; The predicted projection image output module is used to input the projection parameters of the input projection image into the decoder, perform spatial transformation on the projection field of the machine learning representation of the 3D reconstructed target structure inside the decoder, and output the predicted projection image; wherein, the machine learning representation includes an explicit representation and an implicit representation; the explicit representation of the 3D reconstructed target structure includes: encoding the Coulomb potential value of the 3D structure through the grid points of the 3D grid matrix; the implicit representation of the 3D reconstructed target structure includes: using a neural network to represent the function that maps spatial coordinates to the Coulomb potential value of the 3D structure; A mask processing module is used to perform mask processing on the machine learning representation of the 3D reconstructed target structure, the predicted projection image, and the input projection image through the mask; The loss value calculation module is used to calculate the loss value based on the input projection image and the predicted projection image after the mask processing, and to adjust the parameters of the decoder based on the loss value; The decoder training completion module is used to repeatedly execute the steps of obtaining the input projection image of the cryo-electron microscope and the label information of the input projection image from the sample set until the preset training conditions are met, and the trained decoder is obtained.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the training method of the cryo-electron microscopy real-space decoder according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the training method of the cryo-electron microscope real-space decoder according to any one of claims 1 to 9.

Images