Freeze electron microscope tomography imaging reconstruction method and system based on generative adversarial network

Abstract

The application discloses a cryogenic electron microscope tomography imaging reconstruction method and system based on a generative adversarial network, and an electronic device. The method comprises the following steps: initializing an energy model and a generative model, jointly training the energy model and the generative model, and iteratively updating model parameters of the energy model and the generative model; when the energy model and the generative model converge, a trained generative model is obtained; an electron microscope tomography image is cropped into a plurality of overlapping sub-tomography images, a plurality of refined images are generated based on the trained generative model, and the plurality of refined images are recombined. The scheme improves the tomography imaging reconstruction quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a CryoET imaging reconstruction method and system based on a generative adversarial network and an electronic device. BACKGROUND

[0002] CryoET (Cryo Electron Tomography) technology is an important means for in-situ structural analysis of biological samples such as cells, subcellular structures, viruses, organelles, and macromolecular polymers. However, due to objective physical constraints, such as the ability to collect images only within a range of ±60°, CryoET faces a serious problem of wedge-shaped missing in the frequency domain of the reconstructed structure, which leads to anisotropic resolution and sample blurring in the reconstructed results, and is not conducive to the progress of scientific discovery.

[0003] To solve this problem, the industry currently mainly uses IsoNet and its derivative methods. However, the 3D reconstruction problem of CryoET is a typical ill-posed inverse problem. IsoNet ignores this important premise when solving this problem and forcibly assumes that there is a one-to-one mapping between the real sample and the reconstruction model. Training based on this assumption can lead to a pattern collapse problem, which reduces the reconstruction quality. SUMMARY

[0004] To solve the problem of low reconstruction quality in the prior art, the present application provides the following technical solutions.

[0005] The first aspect of the present application provides a CryoET imaging reconstruction method based on a generative adversarial network, comprising:

[0006] initializing an energy model and a generative model, jointly training the energy model and the generative model, and iteratively updating model parameters of the energy model and the generative model, and obtaining a trained generative model when the energy model and the generative model converge;

[0007] cropping the CryoET image into a plurality of overlapping sub-tomographic images, generating a plurality of refined images based on the trained generative model, and recombining the plurality of refined images.

[0008] Preferably, the energy model is defined as:

[0009]

[0010] wherein, p y is the distribution of y; σ 2 , σ 2 g is the noise level, and σ 2n to estimate the noise variance, ∈, ∈ g is randomly generated noise; p ∈ is the distribution of ∈, N is a Gaussian function, and I represents an identity matrix; represents expectation, E() is an energy operation, and R represents a rotation operation selected from a predefined set of rotations, represents a set of R; T M is a tomographic imaging operator; represents a combination of functions; g θ is a generative model, θ is a neural network parameter of the generative model, and φ is a neural network parameter of the energy model.

[0011] Preferably, the joint training of the energy model and the generative model and the iterative updating of the model parameters of the energy model and the generative model are performed until the energy model and the generative model converge, and a trained generative model is obtained. Further comprising:

[0012] Let

[0013] Update

[0014] Update

[0015] Reduce the penalty term λ, and assign the value of θ to θ', until convergence, and return the trained generative model g θ .

[0016] Preferably, after the recombination of the plurality of refined images, further comprising:

[0017] averaging the overlapping regions using a weighted process.

[0018] The second aspect of the present application provides a cryo-electron microscope tomographic imaging reconstruction system based on a generative adversarial network, comprising:

[0019] a training unit configured to initialize an energy model and a generative model, perform joint training of the energy model and the generative model, and iteratively update model parameters of the energy model and the generative model, until the energy model and the generative model converge, and obtain a trained generative model;

[0020] an inference unit configured to crop an electron microscope tomographic image into a plurality of overlapping sub-tomographic images, generate a plurality of refined images based on the trained generative model, and recombine the plurality of refined images.

[0021] The third aspect of the present application provides an electronic device, comprising a processor and a memory, the memory stores a plurality of instructions, and the processor is configured to read the instructions and perform the method provided in the first aspect.

[0022] The fourth aspect of the present application provides a computer readable storage medium, which stores a plurality of instructions, the plurality of instructions can be read and executed by a processor to perform the method provided in the first aspect.

[0023] The beneficial effects of the present application are: the energy model is used to model the data distribution, which avoids the training instability caused by the problem of taking the average among multiple local minimum values, the random noise is introduced to solve the one-to-many mapping problem, the energy model and the generative adversarial network are combined, and the energy function does not need to be explicitly defined. Diversified and reasonable reconstruction results can be generated. The training process is more stable, and recursive updating is not required, which accelerates the training progress. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The flowchart of the IsoNet reconstruction method according to the prior art.

[0025] Figure 2 The flowchart of the cryo-electron microscope tomography imaging reconstruction method based on the generative adversarial network according to the present application.

[0026] Figure 3 The architecture diagram of the cryo-electron microscope tomography imaging reconstruction method based on the generative adversarial network according to the present application.

[0027] Figure 4 The schematic diagram of the purified ribosome corrected by IsoNet.

[0028] Figure 5 The schematic diagram of the central Fourier slice of the purified ribosome corrected by IsoNet.

[0029] Figure 6 The schematic diagram of the corrected purified ribosome according to the scheme of the present application.

[0030] Figure 7 The schematic diagram of the central Fourier slice of the corrected purified ribosome according to the scheme of the present application. DETAILED DESCRIPTION

[0031] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0032] The method provided in the application can be implemented in a terminal environment which can include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction which is loaded and executed by the processor to implement the cryo-electron microscope tomography imaging reconstruction method based on a generative adversarial network described in the embodiments below. The processor can include one or more processing cores. The processor connects various parts in the entire terminal through various interfaces and lines, executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. The memory can include random access memory (RAM) and read-only memory (ROM). The memory can be used to store instructions, programs, codes, code sets or instructions. The display screen is used to display the user interface of various application programs.

[0033] In addition, those skilled in the art can understand that the structure of the terminal described above does not constitute a limitation on the terminal, and the terminal can include more or fewer components, or combine certain components, or different component arrangements. For example, the terminal also includes radio frequency circuitry, an input unit, a sensor, audio circuitry, a power supply, and the like, which are not described here.

[0034] Embodiment one

[0035] To address the shortcomings of the prior art, the application uses a Bayesian framework for in-depth analysis, uses an energy model to model the distribution of the reconstruction result, ensures that the structure recovered in each reconstruction process falls in the low-energy region, avoids mode collapse, and improves the reconstruction quality. The overall scheme includes three main stages: first, define the problem from the perspective of probability modeling; second, design an energy model E to capture the distribution of the missing wedge region; and third, train a neural network prediction model g θ to generate a complete tomographic image by combining the missing wedge input with the energy model.

[0036] As shown in Figure 1 , the embodiment of the application provides a cryo-electron microscope tomography imaging reconstruction method based on a generative adversarial network, which specifically includes:

[0037] Step 101, initialize the energy model and the generative model, jointly train the energy model and the generative model, and iteratively update the model parameters of the energy model and the generative model, and when the energy model and the generative model converge, a trained generative model is obtained.

[0038] The complete algorithm includes two stages, namely training and inference. First, the generative model g θto minimize a pre-trained energy model. Then, the generative model is used to fill in the missing wedges in the tomographic image.

[0039] In the training phase, first input the tomographic image dataset Y, noise level σ 2 , σ 2 g , σ 2 h > 0, estimate the noise variance penalty term λ > 0, energy model E φ ; generative model g θ , learning rate η. y is the observation value of x where the energy model E φ and the generative model g θ are models learned by neural networks. N is a Gaussian function, and I represents a unit matrix.

[0040] Randomly generate noise

[0041] The energy model E φ can be derived by a generative adversarial network. Specifically, to train the energy model E φ and g θ simultaneously, the energy model E φ is defined as:

[0042]

[0043] θ is the neural network parameter of the generative model, and φ is the neural network parameter of the energy model; p y is the distribution of y; p ∈ is the distribution of ∈, E() is the energy operation, and

[0044]

[0045] Update

[0046] Update

[0047] Reduce the penalty term λ, and assign the value of θ to θ'. Until convergence, and return the trained generative model g θ . represent the expectation. R represents a selected rotation operation in a predefined rotation set, represent the set of R. φ, θ are neural network parameters.

[0048] The present application integrates two parts to train the model of the present application, including consistency loss and posterior maximization. The consistency loss is used to ensure that g θ is T Mthe approximate inverse function of T; maximum a posteriori is used to maximize the log-likelihood of the generated result log p(x|y) by an energy penalty term.

[0049] The desired source space is defined as X = {x e X | T M (x) e Y}, the goal is to find g θ x is the original sample data, Y is the observation space, T M is the tomographic imaging operator. However, due to the lack of ground truth data, the energy model cannot be trained directly on the domain X, but only on the observation space Y. Therefore, if a generated sample x lies in X, its energy E(M(x)) should be low. Then the distribution pyis defined by the Boltzmann distribution using the trained energy model.

[0050] The energy function is learned in a way similar to contrastive learning, assigning low energy to samples and high energy to other regions.

[0051] It can be assumed that g θ is the approximate inverse function of T M Therefore, a consistency loss is first introduced to guarantee the inverse condition:

[0052]

[0053] where denotes the composition of functions. To solve the problem of non-injective mapping during training, the penalty term for forced reconstruction can be gradually reduced, λ.

[0054] Next, the result that maximizes the log-likelihood of the posterior log p(x|y) is generated by introducing an energy penalty term. The goal is to minimize the error on X using the model trained on Y. Therefore, it must be ensured that E(T M (x)) is low when T M (x) e Y, and high otherwise. Furthermore, it is assumed that if x e X, then R(x) e X as well, where R denotes a rotation operation selected from a pre-defined set of rotations. Therefore, the posterior is defined as:

[0055]

[0056] where is an introduced hyperparameter.

[0057] Energy-based models are a probabilistic framework that uses the concept of energy to model machine learning problems. An energy function assigns lower energy values to more probable or preferred configurations, and higher energy values to less probable or less desirable configurations. The goal of this system is to identify the state that minimizes the energy and form the energy landscape accordingly.

[0058] Let y be drawn from the distribution p y∈ is a density of p ∈ The distribution p is absolutely continuous. y+∈ It is also an absolutely continuous density:

[0059]

[0060] Where y is the set of observed values ​​y.

[0061] Step 102: Crop the electron microscope tomographic image into multiple overlapping sub-tomographic images, generate multiple thinned images based on the trained generative model, and recombine the multiple thinned images.

[0062] During the inference phase, the complete tomographic image is first cropped into multiple overlapping sub-tomographic images, and then processed via g... θ Multiple sub-tomographic images are processed to produce a refined sub-tomographic image. The refined sub-tomographic images are then recombined into a complete tomographic image, and an averaging process is used to average overlapping areas to mitigate edge artifacts.

[0063] To validate the advantages of the framework presented in this application, the well-known Cryo-ET particle selection benchmark, specifically a dataset of purified ribosomes, and a dataset of virus-like particles of immature HIV-1 in single-particle and tomographic reconstructions, are used. Experiments are conducted on calibrated purified ribosomes and compared with IsoNet, including their corresponding central Fourier slices. See [link to relevant documentation]. Figures 4-7 The image corrected by this application exhibits higher contrast and reduced high-frequency features. While both methods effectively fill in the missing wedge-shaped region, IsoNet correction shows an irregular distribution of high-frequency components in the central region, while this application achieves a more consistent distribution. Compared to the irregular distribution of IsoNet, this application minimizes ringing effects and achieves significantly higher contrast while providing better compensation for the missing wedge. Furthermore, this application completes the process in just two hours, compared to 20 hours of runtime for IsoNet on an NVIDIA V100.

[0064] By employing an energy model to model the data distribution, the training instability caused by averaging across multiple local minima is avoided. Random noise is introduced to address the one-to-many mapping problem. Combining the energy model with generative adversarial networks eliminates the need for explicitly defining an energy function. This approach generates diverse and reasonable reconstruction results. The training process is more stable, eliminating the need for recursive updates and accelerating the training process.

[0065] Example 2

[0066] Another aspect of the present application also includes a functional module architecture corresponding to the foregoing method flow, that is, the embodiment of the present application also provides a cryo-electron microscope tomography imaging reconstruction system based on a generative adversarial network, comprising:

[0067] a training unit configured to initialize an energy model and a generative model, jointly train the energy model and the generative model, and iteratively update model parameters of the energy model and the generative model, and obtain a trained generative model when the energy model and the generative model converge;

[0068] a reasoning unit configured to crop an electron microscope tomography image into a plurality of overlapping sub-tomography images, generate a plurality of refined images based on the trained generative model, and recombine the plurality of refined images.

[0069] The system can be implemented by the cryo-electron microscope tomography imaging reconstruction method based on the generative adversarial network provided in the foregoing embodiment one, and the specific implementation method can be referred to the description in the embodiment one, which will not be described here.

[0070] The present application also provides a memory storing a plurality of instructions for implementing the cryo-electron microscope tomography imaging reconstruction method based on the generative adversarial network as described in the embodiment one.

[0071] The present application also provides an electronic device comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor to enable the processor to perform the cryo-electron microscope tomography imaging reconstruction method based on the generative adversarial network as described in the embodiment one.

[0072] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A cryo-electron microscope tomography imaging reconstruction method based on a generative adversarial network, characterized in that, The method comprises the following steps: initializing an energy model and a generation model, jointly training the energy model and the generation model, and iteratively updating model parameters of the energy model and the generation model, and obtaining a trained generation model when the energy model and the generation model converge; cropping the electron microscope tomography image into a plurality of overlapping sub-tomography images, generating a plurality of refined images based on the trained generation model, and recombining the plurality of refined images; The energy model E φ is defined as: wherein, σ 2 , σ 2 g is a noise level, ∈, ∈ g is a randomly generated noise; N is a Gaussian function, I denotes a unit matrix; represents expectation, R denotes a rotation operation selected from a predefined set of rotations, represents a set of R; T M is a tomographic imaging operator; denotes a combination of functions; x is original sample data; g θ denotes a generative model; p y is a distribution of y; p ∈ is a distribution of ∈; E() is an energy operation; θ is a neural network parameter of the generative model; φ is a neural network parameter of the energy model; The jointly training the energy model and the generation model and the iteratively updating model parameters of the energy model and the generation model, and obtaining a trained generation model when the energy model and the generation model converge, further comprises: Set update update η denotes a learning rate, a penalty term λ is reduced, the value of θ is assigned to θ', until convergence, and the trained generative model g is returned θ .

2. The generative adversarial network-based cryo-electron microscope tomography imaging reconstruction method of claim 1, wherein, After the recombining the plurality of refined images, the method further comprises: averaging the overlapping regions using a weighting process.

3. A cryo-electron microscope tomography imaging reconstruction system based on a generative adversarial network, characterized in that, The method comprises the following steps: a training unit configured to initialize an energy model and a generation model, jointly train the energy model and the generation model, and iteratively update model parameters of the energy model and the generation model, and obtain a trained generation model when the energy model and the generation model converge; an inference unit configured to crop an electron microscope tomography image into a plurality of overlapping sub-tomography images, generate a plurality of refined images based on the trained generation model, and recombine the plurality of refined images; The energy model E φ is defined as: where: σ 2 , σ 2 g is the noise level, ∈, ∈ g is a randomly generated noise, N is a Gaussian function, and I represents a unit matrix; represents expectation, R represents a rotation operation selected from a predefined set of rotations, represents a set of R; T M is a tomographic imaging operator; represents a combination of functions; x is the original sample data; g θ represents a generative model; p y is the distribution of y; p ε is the distribution of ε; E() is an energy operation; θ is a neural network parameter of the generative model; φ is a neural network parameter of the energy model; The training unit is further configured to: Set update update η denotes a learning rate, a penalty term λ is reduced, the value of θ is assigned to θ', until convergence, and the trained generative model g is returned θ .

4. The generative adversarial network-based cryo-electron microscope tomography imaging reconstruction system of claim 3, wherein, The inference unit is further configured to: average the overlapping regions using a weighting process.

5. An electronic device, comprising: The method comprises the following steps:

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, which can be read and executed by a processor to perform the cryo-electron microscope tomography imaging reconstruction method based on the generative adversarial network according to any one of claims 1-2. The computer readable storage medium stores a plurality of instructions, which can be read and executed by a processor to perform the cryo-electron microscope tomography imaging reconstruction method based on the generative adversarial network according to any one of claims 1-2.

Images