Training method and denoising method for a two-dimensional cryo-electron microscopy image denoising model

Abstract

The present invention provides a training method for a two-dimensional cryo-electron microscopy image denoising model. The method includes: S1. Obtain an original image dataset, extract the pure noise region corresponding to each image from the original image dataset to obtain pure noise samples and form a pure noise dataset; S2. Use the pure noise dataset obtained in step S1 to train a generative model to obtain a noise generation model, and use the trained noise generation model to generate multiple new pure noise samples to expand the pure noise dataset to obtain a new pure noise dataset; S3. Transfer the pure noise samples in the new pure noise dataset to the simulated noise-free images to obtain a noise-free-noisy dataset composed of noise-free-noisy samples; S4. Use the noise-free-noisy dataset obtained in step S3 to train a convolutional neural network until convergence.

Description

Technical Field

[0001] The present invention relates to the field of image processing, specifically to the field of cryo-electron imaging technology, and more specifically to a training method and a denoising method for a two-dimensional cryo-electron microscopy image denoising model. Background Art

[0002] Cryo-electron microscopy is a technology widely used in the field of structural biology, which can resolve the high-resolution three-dimensional (3D) structures of proteins and macromolecular complexes from a series of two-dimensional (2D) micrographs. However, since the signal-to-noise ratio (SNR) of the original cryo-electron microscopy images can only reach 0.01 - 0.1, which is the lowest among all imaging fields, this greatly affects the accuracy and efficiency of the downstream analysis of cryo-electron microscopy images and reduces the credibility of structure determination. Therefore, image quality restoration operations are usually required, and then particle picking, structure segmentation, and other cryo-electron microscopy image analysis processes are carried out to achieve high-resolution cryo-electron microscopy 3D reconstruction.

[0003] Currently, there are various traditional methods to improve the contrast of cryo-electron microscopy images and reduce the noise level, such as band-pass filtering, Wiener filtering, and BM3D. The basic idea of band-pass filtering is to directly process the image in the frequency domain and preserve the image information in a specific frequency interval. Since noise is generally high-frequency information, signal enhancement can be achieved to a certain extent through frequency truncation. Wiener filtering is an optimal estimator for stationary processes based on the minimum mean square error criterion. The mean square error between the output of this filter and the desired output is minimized. Therefore, it is an optimal filtering system and can be used to extract signals contaminated by stationary noise. The basic idea of BM3D comes from the observation that there are many similar repetitive structures in natural images. It uses the method of image block matching to collect and aggregate these similar structures, and then performs orthogonal transformation on them to obtain a sparse representation of them. By making full use of sparsity and structural similarity, filtering is performed, and then the processing result is aggregated to the position of the original image block. The denoising of BM3D can fully preserve the structure and details of the image and obtain a good signal-to-noise ratio. However, traditional image restoration algorithms need to introduce image prior knowledge to repair lost and degraded information, usually using knowledge summarized from natural images. However, the noise model in cryo-electron microscopy images is usually unknown and very complex, and the noise model will vary due to different experimental parameters. Even for the same batch of data, the noise models between each image will also vary. Therefore, the predefined image prior knowledge used in these traditional methods cannot correctly fit the noise model in cryo-electron microscopy, resulting in limited performance when traditional denoising methods are applied to cryo-electron microscopy data.

[0004] In recent years, deep learning-based denoising methods have shown their advantages. Benefiting from the powerful learning ability of convolutional neural networks (CNNs), many researchers have proposed many related techniques based on CNNs. For example, an encoder-decoder framework with symmetric convolutional and transposed convolutional layers has been proposed for image restoration; a generative adversarial network (GAN) for image super-resolution (SR) has been proposed, which can restore realistic textures from a large number of downsampled images. However, most existing deep learning-based methods require paired noise-free and noisy datasets during training, so it is difficult to apply them to the field of cryo-electron microscopy where noise-free images cannot be obtained. To overcome the data limitation of noise-free-noisy image pairs, several denoising methods that learn from paired noisy images or single noisy images have been proposed in recent years. For example, a general machine learning (ML) framework called Noise2Noise (N2N) has been proposed to learn a denoising model from paired noisy images; or a GAN-CNN-based framework GCBD has been proposed to learn a denoising model from a single noisy image, where GAN is used to construct a paired training dataset and then a convolutional neural network is used for denoising. It is worth noting that a denoiser called Topaz-Denoise, which is based on the N2N architecture and trained using thousands of images, is used to denoise cryo-electron microscopy images and cryo-electron tomography images and is currently the most commonly used deep learning-based cryo-electron microscopy image denoising algorithm. However, as can be seen from the above introduction, most deep learning-based denoising algorithm methods require paired noise-free and noisy datasets during training, so they cannot be applied to the field of cryo-electron microscopy where noise-free images cannot be obtained. Topaz-Denoise is currently the most commonly used deep learning-based cryo-electron microscopy denoising algorithm, which solves the need for paired supervised datasets and uses paired noisy data pairs to train the denoising network model. However, this method has two problems: (1) The framework provides a general model that does not require users to retrain the network model and can be used directly. However, since it is a fully automatic model, when processing a new dataset, if the noise model of the new dataset is quite different from the noise model in the model's training set, then the denoising performance will be greatly affected. (2) The framework can be retrained for a new dataset, and the training set requires paired noisy images. For cryo-electron microscopy images, the framework proposes that paired noisy images can be obtained by splitting the original imaging frames into single and double frame sets, and the single and double frame sets are respectively superimposed to obtain two noisy images with the same signal distribution, so that the training set of the denoising model can be obtained. However, this will limit the use of public datasets or other people's datasets by users because the original imaging frame sequences are often not publicly available. Summary of the Invention

[0005] Therefore, the object of the present invention is to overcome the defects of the above-mentioned prior art and provide a training method and a denoising method for a two-dimensional cryo-electron microscopy image denoising model based on noise modeling and migration.

[0006] According to the first aspect of the present invention, there is provided a training method for a two-dimensional cryo-electron microscopy image denoising model, the method comprising: S1, obtaining an original image data set, extracting a pure noise region corresponding to each image from the original image data set to obtain pure noise samples and forming a pure noise data set; S2, using the pure noise data set obtained in step S1 to train a generative model to obtain a noise generation model, and using the trained noise generation model to generate a plurality of new pure noise samples to expand the pure noise data set to obtain a new pure noise data set; S3, migrating the pure noise samples in the new pure noise data set to a simulated noise-free image to obtain a noise-free-noisy data set composed of noise-free-noisy samples; S4, using the noise-free-noisy data set obtained in step S3 to train a convolutional neural network until convergence.

[0007] Preferably, step S1 includes: S11, generating a pair of simulated noise-free-noisy samples based on real imaging parameters using a simulation tool to form a simulated noise-free-noisy data set; S12, using the simulated noise-free-noisy data set to train a convolutional neural network until convergence to obtain a coarse-grained denoiser; S13, using the coarse-grained denoiser to perform preliminary denoising on the images in the original image data set to enhance the contrast and obtain a contrast-enhanced data set; S14, extracting the pure noise regions in each image of the contrast-enhanced data set.

[0008] In some embodiments of the present invention, in step S11, InSilicoTEM is used to simulate and generate a pair of simulated noise-free-noisy samples, wherein the simulation conditions of InSilicoTEM are set using the experimental parameters in the real image data acquisition process.

[0009] Preferably, in step S12, a convolutional neural network with a U-net architecture is trained using the simulated noise-free-noisy data set until convergence to obtain a coarse-grained denoiser. In some embodiments of the present invention, the convolutional neural network with a U-net architecture includes five max-pooling downsampling layers and five nearest-neighbor upsampling layers, and there is a skip link between downsampling and upsampling at each spatial resolution level. By using the simulated noise-free-noisy data set to train the convolutional neural network with a U-net architecture to learn the optimal denoising function, a coarse-grained denoiser is obtained, wherein the loss function of the convolutional neural network with a U-net architecture is:

[0010] argmin θ E x～X [‖f θ (z)-y‖ p

[0011] Among them, f is the denoising function, θ is the parameter of the denoising function, x is the image with noise y, y is the noise-free image, and X is the dataset of images with noise.

[0012] Preferably, step S14 includes performing the following operations on each image in the contrast-enhanced dataset: S141. Divide the image into an overlapping set of image patches according to a preset patch size and a preset stride; S142. Divide each image patch into a preset number of local patches to form a set of local patches, and calculate the structural similarity between any two local patches in the set of local patches. Among them, the image patches with the structural similarity between any two local patches greater than the preset similarity threshold are the pure noise regions in the original image; S143. Locate the position of the pure noise region in the original image based on the coordinates of the pure noise region in the contrast-enhanced image to extract the pure noise region in the original image. Among them, the preset patch size is 320*320 or 640*640, the preset stride is half of the preset patch size, and the preset number is 4. The preset similarity threshold is 0.7.

[0013] In some embodiments of the present invention, in step S2, a noise generation model is obtained by training a GAN model using a pure noise dataset. The GAN model includes: a generative network and a discriminative network; among them, the generative network includes five fully connected layers, and the discriminative network includes three fully connected layers; both the generative network and the discriminative network use the LeakyReLU activation function, and the loss function of the GAN model is:

[0014]

[0015] Among them, D(*) is the discriminative network function, P r is the distribution of the pure noise dataset, P g is the distribution of the generative network, is the distribution uniformly sampled along a straight line between the paired points sampled from P r and P g , and λ is the hyperparameter of the GAN framework. Among them, the hyperparameter of the GAN model is set to 10.

[0016] In some embodiments of the present invention, in step S3, the pure noise samples generated by the noise generation model and the noise-free samples in the simulated noise-free-noise samples generated by the simulation tool are reweighted with reference to the simulated noise samples corresponding to the noise-free samples to obtain a new set of noise samples represented by the reweighted results of the pure noise samples and the noise-free samples, and form a new paired noise-free-noise sample with the noise-free samples to achieve noise migration, where:

[0017] ​The new noisy samples are represented as:

[0018]

[0019] The goal of noise migration is:

[0020]

[0021] Wherein, is the modulation function, α, β, and γ are scalar coefficients, d is the image block size, {V i} is the pure noise samples generated by the noise generation model, {S i} is the noise-free image generated by the simulation tool, and {X i} is the noisy image corresponding to the noise-free image generated by the simulation tool.

[0022] In some embodiments of the present invention, the step S4 includes: training a coarse-grained denoiser using the noise-free-noisy dataset obtained in step S3 to fine-tune the coarse-grained denoiser to obtain a denoising model. Preferably, the Adagrad optimizer is used to train the model parameters during the process of obtaining the coarse-grained denoiser and fine-tuning to obtain the denoising model, and the learning rate is 0.001.

[0023] According to the second aspect of the present invention, there is provided a method for denoising two-dimensional cryo-electron microscopy images, the method including: T1, obtaining a two-dimensional cryo-electron microscopy image to be processed; T2, performing denoising processing on the two-dimensional cryo-electron microscopy image using the denoising model trained by the method described in the first aspect of the present invention.

[0024] Compared with the prior art, the advantages of the present invention are as follows: The present invention performs denoising according to the true statistical characteristics of the noise in the cryo-electron microscopy image, realizes accurate modeling and migration of the noise in the two-dimensional cryo-electron microscopy image, and is no longer limited by the general noise model, thereby realizing the removal of noise and the restoration of the sample signal. Among them, the present invention uses the pure noise region in the image and the modeling method of deep learning technology to accurately model the noise. This method distinguishes the background region from the sample region based on image similarity, thereby realizing the extraction of the pure noise image region, and combines the generative model in the field of deep learning to complete the accurate modeling of the noise in the cryo-electron microscopy field, which is not limited by a single noise model and can perform accurate modeling according to different noise models. The present invention adopts a reweighting scheme for signals and noise, superimposes the noise-free signal and the true noise distribution together to obtain the noisy data. Thereby, the construction of a supervised denoising dataset is realized, making it possible to train a supervised deep learning denoising model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The following further describes the embodiments of the present invention with reference to the accompanying drawings, wherein:

[0026] Figure 1 Schematic diagram of the principle of the training method for the two-dimensional cryo-EM image denoising model according to an embodiment of the present invention;

[0027] Figure 2 Schematic diagram of the main process of the training method for the two-dimensional cryo-EM image denoising model according to an embodiment of the present invention;

[0028] Figure 3 Schematic diagram of the change process of image data in the training method for the two-dimensional cryo-EM image denoising model according to an embodiment of the present invention;

[0029] Figure 4 Schematic diagram of the U-net network architecture adopted in the training method for the two-dimensional cryo-EM image denoising model according to an embodiment of the present invention;

[0030] Figure 5 Schematic diagram of the GAN model structure adopted in the training method for the two-dimensional cryo-EM image denoising model according to an embodiment of the present invention;

[0031] Figure 6 Schematic diagram of the denoising result of the denoising model trained by the training method for the two-dimensional cryo-EM image denoising model according to an embodiment of the present invention on the T20S proteasome data;

[0032] Figure 7 Schematic diagram of the denoising result of the denoising model trained by the training method for the two-dimensional cryo-EM image denoising model according to an embodiment of the present invention on the structural data of the binding of Plasmodium falciparum 80s ribosome and an anti-protozoal drug; Detailed implementation manners

[0033] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0034] It should be noted that the original cryo-EM images, original images, and original noisy image patches mentioned in the following embodiments all refer to the original un-denoised cryo-micrographs. In some places, for the convenience of description, the preliminarily denoised image patches, preliminarily denoised cryo-EM images, and preliminarily denoised two-dimensional cryo-EM images all refer to the images obtained by preliminarily denoising the original cryo-micrographs with a coarse-grained denoiser. The pure noise patches and pure noise regions both refer to pure noise data.

[0035] First, the basic idea and principle of the present invention will be introduced.

[0036] As described in the background art, cryo-electron microscopy is a widely used technique for ultrastructure determination, which constructs the three-dimensional structures of proteins and macromolecular complexes from a set of two-dimensional micrographs. However, limited by the electron beam dose, the micrographs in cryo-electron microscopy usually suffer from extremely low signal-to-noise ratios, which affects the efficiency and effectiveness of downstream analysis. In particular, the noise in cryo-electron microscopy is not simple additive or multiplicative noise, and its statistical properties are completely different from those of natural images, which greatly affects the performance of traditional noise reduction methods. Therefore, there is a great need for effective denoising algorithms in the field of two-dimensional cryo-electron microscopy images.

[0037] When conducting research on the denoising algorithm for cryo-electron microscopy images, the inventors found that there are mainly two problems in existing deep learning methods:

[0038] First, existing methods fail to accurately estimate the noise model in electron microscopy images. Due to the complexity of the cryo-electron microscopy imaging principle and the influence of various factors during the data acquisition process, it is difficult to accurately describe the noise model in the image with a mathematical formula, which also limits the performance of existing denoising methods based on prior knowledge of noise.

[0039] Second, there is a lack of paired supervised noiseless-noisy image training sets. Deep learning methods have powerful learning capabilities and can learn the noise characteristics from a large amount of data, so that they can complete noise removal without prior knowledge of noise statistical characteristics. However, there are no noiseless images in the field of cryo-electron microscopy, which limits the learning-based methods. Although there are denoising algorithms based on paired noisy images and single noisy images, the denoising performance of these algorithms is limited in principle due to the lack of noiseless images, and the acquisition of data is not simple.

[0040] During the research process, the inventors found that there are pure noise regions in cryo-electron microscopy images (hereinafter referred to as cryo-EM images, and the following descriptions will all be based on cryo-EM images). Therefore, deep learning technology and the pure noise regions in cryo-EM images can be used to model the noise. Since the noise model in cryo-EM images is too complex, and various experimental parameter changes during the electron microscopy imaging process will cause changes in the noise model, and even the noise between images in the same batch will also vary. Such abstract features can be extracted using deep learning technology with powerful modeling capabilities, but deep learning requires a large number of training sets. Considering the characteristics of cryo-EM images, as long as the appropriate size is selected, the regions in the electron microscopy images can be divided into two types of regions: pure noise regions and regions containing biological sample signals. In this way, many pure noise image patches can be obtained, and combined with deep learning technology, an accurately estimated noise model can be obtained. As mentioned in the background technology, the main factor currently restricting the performance of deep learning-based denoising algorithms in the field of cryo-EM is the lack of supervised noise-free - noisy image training sets. Therefore, the present invention uses the means of noise migration to migrate the noise to a simulated noise-free image with the same real imaging parameters to generate a paired noise-free - noisy training set with a real noise distribution. Specifically, by extracting the pure noise regions in the real image dataset to model the cryo-EM noise, and then migrating the noise characteristics to the noise-free image, a noisy image with specified noise and the corresponding noise-free image are obtained to construct a supervised noise-free - noisy dataset. Based on such a supervised dataset, combined with deep learning, a denoising model for the noise in a specific dataset can be trained and the dataset can be denoised to obtain a denoising result. Generally speaking, the method of the present invention is as Figure 1 shown. Based on the real imaging parameters of real cryo-EM images, noise-free - noisy paired simulation data is generated based on a simulation tool, and thus a simulated dataset is formed to train a coarse-grained denoiser. The coarse-grained denoiser is used to enhance the contrast of real cryo-EM images to obtain a preliminarily denoised microscopic image. The pure noise regions are located in the preliminarily denoised microscopic image to obtain the coordinates of the pure noise patches. Based on these coordinates, pure noise image patches are extracted from the original real cryo-EM images. Based on the pure noise image patches, diverse noise samples are generated through a noise generator, and the noise is migrated to the noise-free image to obtain a noise-free - noisy image pair with a real noise distribution. The noise-free - noisy image pair with a real noise distribution is used to train a convolutional neural network to obtain a fine-grained denoiser, and the fine-grained denoiser denoises the real cryo-EM images to obtain a denoising result.

[0041] The present invention will be described in detail below with reference to the accompanying drawings.

[0042] As described above, the object of the present invention is to solve the problems that the noise model in cryo-EM image denoising is unknown and difficult to estimate, and there is a lack of a supervised paired noiseless-noisy training set that can be used for deep learning. As Figure 2 shown, the present invention provides a training method for a two-dimensional cryo-EM image denoising model. The method includes steps S1, S2, S3, and S4, which are described in detail below.

[0043] In step S1, an original image dataset is obtained, and a pure noise region corresponding to each cryo-EM image is extracted from the original two-dimensional cryo-EM image dataset to obtain pure noise samples and form a pure noise dataset. Since the signal-to-noise ratio of the original cryo-EM image is extremely low, it is difficult to directly distinguish particles and background regions from the original cryo-EM image. Therefore, according to an embodiment of the present invention, a preprocessing, i.e., preliminary denoising, is first performed on the original real cryo-EM image dataset to distinguish particles and background regions (pure noise regions). Considering that there is no completely noiseless image in the field of cryo-EM images, according to an embodiment of the present invention, a simulation tool is used to generate a simulation dataset according to the real cryo-EM image imaging parameters for training a coarse-grained denoiser, and the trained coarse-grained denoiser is used to perform contrast enhancement on the preliminary denoising of the original real cryo-EM image dataset, so as to more easily locate the pure background region and particles in the real cryo-EM image, locate the coordinates of the pure noise blocks based on block similarity, and use the located coordinates to extract the pure noise regions from the original real cryo-EM image dataset. According to an embodiment of the present invention, step S1 includes the following steps:

[0044] Step 101: Generate simulation data based on real imaging parameters. As mentioned in the background art, the lack of supervised training data hinders the application of deep learning-based noise reduction methods in cryo-EM. To solve this problem, as Figure 3As shown in the section of "a. Noise extraction", first, data simulation based on real experimental parameters is performed, that is, a simulation tool is used to simulate noisy image patches and simulated noise-free image patches based on real imaging parameters. According to an embodiment of the present invention, a paired noise-free-noisy dataset is generated using the simulation software InSilicoTEM, and this simulation data generation software simulates the entire imaging process of a cryo-electron microscope based on physical principles. According to an embodiment of the present invention, the simulation dataset generated by the simulation data generation software InsilicoTEM according to the real cryo-electron microscope image imaging parameters includes paired noise-free-noisy images, and has the same imaging parameters as the real cryo-electron microscope image dataset. Therefore, the simulated images in the simulation dataset have a similar noise distribution to the real cryo-electron microscope images. Among them, the simulation conditions are set using the experimental parameter settings during the acquisition of the real dataset, including pixel size, defocus, voltage, electron dose, and detector type, etc. In this way, the simulation data can be very close to the real data in terms of resolution limit, contrast transfer function, and modulation effect. In addition, according to an embodiment of the present invention, a homologous protein similar in size to the real sample is downloaded from the Protein Data Bank (PDB) as the sample of the simulation data, and such a simulation is likely to generate a dataset with noise statistical characteristics similar to those of the real data.

[0045] Step 102: Train a coarse-grained denoiser. As described above, in step 101, a noise-free-noisy simulation dataset composed of paired noise-free-noisy simulation image pairs can be generated, as Figure 3 shown in the section of "a. Noise extraction" in []. The noise-free-noisy simulation dataset is input into a convolutional neural network, that is, using this simulation dataset to train a CNN can obtain a coarse-grained denoiser (also called a sparse CNN denoiser), which can realize the contrast enhancement of the original real cryo-electron microscope image dataset. This is because the training set generated by simulation has the same imaging parameters as the original real cryo-electron microscope image dataset. Therefore, this model can achieve a certain level of noise reduction and contrast enhancement on real data to simplify the distinction between the background and particles and realize the extraction of noise patches.

[0046] According to an embodiment of the present invention, the coarse-grained denoiser of the present invention is based on the U-net architecture, as Figure 4 shown, which includes five max-pooling downsampling modules and five nearest-neighbor upsampling layers, and there is a skip connection between downsampling and upsampling at each spatial resolution level. Since the U-net architecture is a commonly used technology in the art, the structural details thereof will not be described herein again, and the principle of finding the denoising function will be mainly described. Given a series of noise-free-noisy image pairs {noise-free image y - noisy image x ∼ Noise(y)}, a denoising function f with parameters θ can be learned. The loss function of the task is:

[0047] argmin θ E x～X [‖f θ (x)-y‖ p

[0048] where X is a noisy image dataset, p is the p-norm. According to an embodiment of the present invention, the present invention uses p = 2 to find the optimal f denoiser.

[0049] Step 103: Extraction of pure noise blocks based on image patch similarity. If a cryo-EM image is divided into small patches of appropriate size, these patches can be classified into two categories: patches containing biological sample signals or background patches with pure noise. The distributions among all the background patches are very similar, while the patches containing signals have different signal patterns. Since the ice in cryo-EM imaging is almost transparent, the background image patches can represent pure noise regions. To distinguish whether an image patch is a pure noise region or a region containing signals, according to an embodiment of the present invention, a pure noise extraction method based on image patch similarity is adopted to extract the pure noise regions in the cryo-EM image. As Figure 3 shown in the "a. Noise extraction" section in, the present invention uses the trained coarse-grained denoiser to preliminarily denoise the original real cryo-EM image to obtain preliminarily denoised image patches and locates the pure noise patches therein based on patch similarity, and then extracts pure noise patches (the regions corresponding to the pure noise patches are pure noise regions) from the original noisy image patches based on the located coordinates to obtain a pure noise dataset composed of pure noise samples.

[0050] Since the signal-to-noise ratio in cryo-EM micrographs is extremely low, it is a difficult task to distinguish the background from the sample signals in the original noisy micrographs. Here, to extract the pure noise regions, two steps are required. First, the present invention trains a coarse-grained denoiser to roughly enhance the contrast of the EM image and help extract the noise regions. Next, using the preliminarily denoised image, a noise extraction scheme based on patch similarity, if a cryo-EM image is divided into small patches of appropriate size, these patches can be classified into two categories: patches containing biological sample signals or background patches with pure noise. The distribution in the background region is uniform, while the image regions containing biological signals have different patterns. Since the ice used in the process of preparing samples for EM imaging is almost transparent, the background region can be considered as a pure noise region. Thus, by calculating the similarity of the sub-regions inside each patch, it can be determined whether the current image region is a pure noise region.

[0051] According to an embodiment of the present invention, step 103 includes the following sub-steps:

[0052] ​Step 1031: Given the micrograph I in the dataset, use the coarsely grained denoiser trained in Step 102 to preliminarily denoise the electron microscope image to obtain a contrast-enhanced image I′;

[0053] Step 1032: Divide the contrast-enhanced image I′ into an overlapping block set Θ = {P i}, and the size of each image block is d×d pixel 2 , and the step size is set to s. Among them, the block size d is usually set to 320 or 640, and the step size s is set to half of the block size.

[0054] Step 1033: For any image block P i ∈Θ, further divide P i into N local blocks {P i，k}, and calculate the structural similarity SSIM (structural similarity) between any two sub-blocks i，k in this set of local block sets {P }, where N is the preset number of local blocks. According to an embodiment of the present invention, N takes the value of 4.

[0055] Step 1034: For each image block P i , if then it can be determined that P i is a background block, that is, a pure noise area, where thre is a given similarity threshold, which is default set to 0.7. According to an embodiment of the present invention, the similarity measurement is calculated as follows:

[0056]

[0057] Among them, and are the mean and standard deviation of , where and are the mean and standard deviation of . is and 's covariance. c1 and c2 are regularization constants, and they have very small values to avoid extremely small denominators. According to the principle of similarity, c1 = (K1L) 2 , c2 = (K2L) 2 , when the image is an 8-bit image, L = 255, and K1 = 0.01, K2 = 0.03 are set. In the present invention, the pixel values of the entire image are regularized to between [0, 1], so the present invention sets L = 1 to achieve the same effect.

[0058] Step 1035: Repeat steps 1031 - 1035 until the background blocks of all the images in the entire dataset are identified, and then extract the noise blocks from the original image I.

[0059] It should be noted that image I′ is a contrast-enhanced version of image I. They correspond to the same signal. However, since preliminary denoising has been performed on image I′, the noise pattern has been damaged, and the real noise needs to be extracted from the original image. All the hyperparameters here (including s, d, N, thre) can be adjusted according to the characteristics of the dataset. It is worth noting that the calculation and determination of similarity are carried out on the denoised image I′, aiming to locate the positions of the pure noise blocks to obtain the positioning coordinates. However, the extraction of the pure noise blocks is based on the coordinates of the located pure noise blocks on the original cryo-EM image I.

[0060] In step S2, use the pure noise dataset obtained in step S1 to train a generative model to obtain a noise generation model, and use the trained noise generation model to generate several new pure noise samples to expand the pure noise dataset to obtain a new pure noise dataset. Generally, in order to save costs, during the data acquisition process of cryo-EM images, without aggregation, the biological samples will be distributed in the images as much as possible. When extracting the pure noise regions, it is hoped that the size of the noise blocks is as large as possible. When corresponding to the original image, the signals of the biological samples can be included in this sized region. Therefore, only a small number of pure noise regions can be extracted from the dataset, which may lack diversity and cannot summarize the statistical characteristics of the noise in the entire dataset. Therefore, the present invention uses a generative model in the field of deep learning to learn the noise distribution, trains a noise generator, and then generates more diverse samples and simultaneously completes data augmentation.

[0061] According to an embodiment of the present invention, the present invention uses a generative model to complete the expansion of the number and pattern of pure noise samples to achieve noise model estimation and data enhancement. The noise model in cryo-EM is too complex to be clearly described by an analytical expression. Therefore, after obtaining the samples of pure noise, the samples are input into a generative model (such as a generative adversarial network) to train an accurate estimate of the noise model, and the trained generative model is used to generate more noise samples (pure noise samples) for constructing the subsequent dataset. According to an embodiment of the present invention, the present invention adopts an improved GAN framework, namely WGAN-GP, to implicitly learn the potential noise model in cryo-EM micrographs. As Figure 3As shown in the section of "b. Noise Modeling", a GAN noise generator is obtained by training a GAN framework with a pure noise data set composed of pure noise samples extracted from the original cryo-EM images. Then, the GAN noise generator generates multiple noise samples that are consistent with the noise distribution of the original images. According to an embodiment of the present invention, the initialization of the network weights of the WGAN-GP network architecture adopted by the GAN framework noise generator uses a standard Gaussian distribution with a standard deviation σ = 0.02. The slopes of the LeakyReLU activation functions used in the generator and the discriminator are both set to 0.2. The Adam optimizer is used for model training, where the hyperparameters of Adam are set to 0.5, the weight update parameter β1 = 0.999, and the learning rate is 0.0002.

[0062] As Figure 5 shown, the GAN framework model includes two components, a generative network composed of five fully connected layers, and a discriminative network composed of three fully connected layers. Batch normalization is used in the generator to ensure the stability of the model, and the LeakyReLU activation function is used in both the generator and the discriminator to ensure fast learning.

[0063] The generative network is trained to generate noise samples, while the discriminative network is trained to determine whether the samples are from real noise samples or fake noise samples generated by the generative network. After the integration of adversarial learning, the generative network will be able to produce noise blocks that are difficult to distinguish from real noise blocks. The loss function of its task is

[0064]

[0065] where D(*) is the discriminator function, P r is the distribution of the noise block data set, P g is the distribution of the generator, is defined as the distribution uniformly sampled along the straight line between the paired points sampled from Pr and Pg. λ is a hyperparameter of the GAN network, generally taking a value of 10. The generative model not only expands the number of noise samples but also expands the diversity of the noise.

[0066] In step S3, pure noise samples in the new pure noise dataset are migrated to the simulated noise-free images to obtain a noise-free-noisy dataset composed of noise-free and noisy samples. The noise generator mentioned in the previous embodiments can generate noise blocks with almost the same statistical characteristics as the real noise in cryo-EM images. After obtaining the noise blocks, the present invention designs a noise and signal reweighting algorithm based on image contrast to migrate the noise generated in the noise synthesizer to the noise-free images to generate noisy data with a specified noise distribution. The noise-free images used here are simulated data. However, since the noise in cryo-EM images is not simple additive or multiplicative noise, the weights between the noise and the signal need to be recalculated. The obtained noise-free-noisy image pairs can be used for the training of the final denoiser.

[0067] After obtaining the noise samples, the noise can be migrated to the noise-free data, such as the simulated dataset generated in step S1, to construct noisy data with a noise distribution consistent with that in the real dataset, thereby obtaining a paired noise-free-noisy image to form a training dataset for implementing signal and noise superposition based on contrast. As Figure 3 shown in the "c. Denoiser training" part of

[0068] , based on the signal and noise superposition algorithm based on image contrast, the noise samples generated by the GAN noise generator are migrated to the noise-free images to obtain a training dataset composed of new noise-free-noisy image pairs. The reason for this is that although the present invention has previously designed a simulation based on experimental parameters to generate simulated clean noise pairs, these simulated data are still insufficient to present the noise pattern in real cryo-EM images because the cryo-EM imaging process is much more complex than the principle of simulated imaging. However, the GAN noise generator can generate noise blocks with almost exactly the same statistical characteristics as the real noise in cryo-EM images. Thus, according to an embodiment of the present invention, a signal and noise superposition algorithm based on image contrast is provided to migrate the noise generated in the GAN noise generator to the simulated noise-free images.

[0068] According to an embodiment of the present invention, the input of the signal and noise superposition algorithm based on image contrast is the pure noise image block {V i} generated in the GAN and the noise-free image block {S i} that only contains the sample signal generated by the simulation tool. Using the noisy simulated image {X i} corresponding to the noise-free signal as a reference, the reweighting result {Y i} of the noise generated by the GAN and the noise-free signal is output. Here, since the experimental parameters used in the generation processes of the simulated noise-free and noisy images are the same as those of the real data, the graphic contrast of {X i} can be used as a reference, and the weighted result is denoted as ( is the modulation function), the noise migration should conform to the following objective function:

[0069]

[0070] where α, β, and γ are scalar coefficients, and d is the image block size. Such a minimization problem can be easily solved by the least squares method. Then, using the solved coefficients, the signal will be reweighted and modulated to generate a noise-free / noise-containing image pair.

[0071] In step S4, the noise-free / noise-containing dataset obtained in step S3 is used to train a convolutional neural network until convergence to obtain a denoising model. According to an embodiment of the present invention, the constructed noise-free / noise-containing image pairs are input into a fine-grained denoiser to train a denoising model that can capture real noise statistics and recover the sample signal from the noise. According to an embodiment of the present invention, in order to save training time, network fine-tuning can be directly performed based on the parameters of the coarse-grained denoiser model instead of retraining. As shown in the "c. Denoiser training" section of Figure 3 , the previously obtained sparse CNN denoiser is fine-tuned using the training dataset to obtain a denoising model, and then the original image is denoised to obtain a denoising result. Preferably, based on the network training mode of fine-tuning the coarse-grained / fine-grained denoiser, there are two denoisers involved in the present invention, the coarse-grained denoiser and the fine-grained denoiser: (i) The coarse-grained denoiser is used to perform preliminary denoising on the original electron microscope image data, roughly enhancing the contrast of the image to facilitate distinguishing the background area and the area where the biological sample is located, so as to extract the pure noise image area. This coarse-grained denoiser is trained using a paired noise-free / noise-containing simulation dataset, and the imaging parameters used for generating the simulation data are the same as those of the real data; (ii) The fine-grained denoiser is the denoiser that can denoise real data ultimately obtained in the present invention, and is trained by the dataset reweighted in step S3 to capture the noise statistical characteristics in the real data and recover the real sample signal. Here, the fine-grained denoiser is obtained by fine-tuning on the weights of the coarse-grained denoiser rather than training from scratch. Since the training set of the coarse-grained denoiser has the same imaging parameters as the real dataset, it can be believed that the coarse-grained training set can provide a relatively accurate initial weight. Among them, the coarse-grained CNN denoiser for enhancing the contrast of real data images is trained using a simulation dataset, and the initial weight is the default value. The fine-grained CNN denoiser for final denoising is trained using the dataset reweighted from pure noise and noise-free images, and the initialization parameter is the weight of the coarse-grained denoiser. Both models use the Adagrad optimizer to train the model parameters, and the learning rate is 0.001.

[0072] To verify the effectiveness of the present invention, specific experimental data will be used for detailed illustration below.

[0073] Experimental materials:

[0074] The inventor used two sets of real datasets: a set of 191 T20S proteasome data (EM25) with a size of 7420*7676 and a set of 600 Plasmodium falciparum 80s ribosome and anti-protozoal drug binding structure data (EM28) with a size of 4096*4096 to verify the actual effectiveness of the present invention.

[0075] First, the method of the present invention was used to train a denoising model with real datasets. Among them, to train the coarse-grained denoiser, the inventor used the cryo-electron microscopy simulation data generation software InsilicoTEM to generate a set of 250 simulated data with the same parameters and a size of 4096*4096. When training the fine-grained denoiser, all electron microscopy images were sliced into image patches with a size of 320*320, the slicing step size was 160, and 10% of the data was randomly selected as the validation set.

[0076] Then, an evaluation method was used to evaluate the effectiveness between the present invention and other different methods. SNR is one of the most commonly used image quality evaluation methods. The present invention selects SNR to measure the performance of different denoising algorithms. The higher the corresponding SNR, the better the denoising performance and effect. Since there is no noise-free image, SNR can only be calculated by estimation. First, 10 paired signal and background regions were selected from 10 electron microscopy images, where the background region was as close as possible to the corresponding signal region. Given N signal and background image pairs, The mean and variance of each background region were marked as The signal of each region was defined as The mean and variance of the signal region were calculated, The average SNR (dB) of this region was defined as:

[0077]

[0078] Figure 6 Shows the denoising effects of different denoising methods mentioned in the background technology and the method of the present invention on EM25, Figure 7 Shows the denoising effects of different denoising methods mentioned in the background technology and the method of the present invention on EM28, where NT2C represents the present invention. From Figure 6 and Figure 7 It can be seen that the denoising effect of the present invention is the best. Table 1 shows the comparison results of the denoising model obtained by the method of the present invention and other methods on the test dataset.

[0079] Table 1

[0080]

[0081] As can be seen from Table 1, the denoising model obtained by the method of the present invention has the best performance.

[0082] From the description of the above embodiments, on the one hand, the present invention uses the pure noise region in the real dataset combined with deep learning technology to complete the accurate modeling of noise. On the other hand, the estimated noise model is migrated to other noise-free images to obtain a noisy image with a real noise distribution, and thus a supervised dataset required for training a deep learning denoising network can be obtained, that is, a paired noise-free - noisy image training set. Since the noise statistical characteristics included in this training set are exactly the same as those of the real dataset, this noise model can denoise the real dataset.

[0083] Compared with the existing deep learning-based cryo-EM image denoising methods, the present invention has the following beneficial effects: 1. The present invention can denoise according to the real statistical characteristics of the noise in cryo-EM images, and the denoising effect is better than that of other current denoising algorithms. The present invention realizes the accurate modeling and migration of the noise in two-dimensional cryo-EM images, thereby realizing the removal of noise and the restoration of sample signals. 2. The present invention proposes a modeling method using the pure noise region in the image and deep learning technology to achieve accurate noise modeling. This method distinguishes the background region from the sample region based on image similarity, thereby realizing the extraction of the pure noise image region, and combines the generative model in the field of deep learning to complete the modeling of the noise in the cryo-EM field. 3. The present invention proposes a signal and noise superposition algorithm based on image contrast to achieve noise migration. In order to migrate the modeled noise to a noise-free image, the present invention designs a reweighting scheme for signals and noise, superimposes the noise-free signal and the real noise distribution together to obtain noisy data. Thus, the construction of a supervised denoising dataset is realized, making it possible to train a supervised deep learning denoising model.

[0084] It should be noted that although the above steps are described in a specific order, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even the order can be changed, as long as the required functions can be achieved.

[0085] The present invention can be a system, a method, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to implement various aspects of the present invention.

[0086] A computer-readable storage medium can be a tangible device that retains and stores instructions for use by an instruction execution device. A computer-readable storage medium may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punched card or raised structures in grooves having instructions stored thereon, and any suitable combination of the foregoing.

[0087] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skill in the art in the technical field to understand the embodiments disclosed herein.

Claims

1. A training method for a two-dimensional cryo-electron microscopy image denoising model, characterized in that The method includes: S1. Obtain an original image dataset, extract the pure noise region corresponding to each image from the original image dataset to obtain pure noise samples and form a pure noise dataset; the step S1 includes: S11. Generate paired simulated noise-free-noisy samples based on real imaging parameters using a simulation tool to form a simulated noise-free-noisy dataset; S12. Train a convolutional neural network with the simulated noise-free-noisy dataset until convergence to obtain a coarse-grained denoiser; S13. Perform preliminary denoising on the images in the original image dataset to enhance the contrast and obtain a contrast-enhanced dataset; S14. Extract the pure noise regions in each image of the contrast-enhanced dataset; S2. Train a generative model with the pure noise dataset obtained in step S1 to obtain a noise generation model, and use the trained noise generation model to generate multiple new pure noise samples to expand the pure noise dataset to obtain a new pure noise dataset; S3. Transfer the pure noise samples in the new pure noise dataset to the simulated noise-free images to obtain a noise-free-noisy dataset composed of noise-free-noisy samples; S4. Train a convolutional neural network with the noise-free-noisy dataset obtained in step S3 until convergence.

2. The method according to claim 1, wherein In step S11, InSilicoTEM is used to simulate the generation of paired simulated noise-free-noisy samples, where the simulation conditions of InSilicoTEM are set using the experimental parameters in the real image data acquisition process.

3. The method according to claim 1, wherein In step S12, a convolutional neural network with a U-net architecture is trained with the simulated noise-free-noisy dataset until convergence to obtain a coarse-grained denoiser.

4. The method according to claim 3, characterized in that, The convolutional neural network with a U-net architecture includes five max-pooling downsampling layers and five nearest-neighbor upsampling layers. There is a skip connection between downsampling and upsampling at each spatial resolution level. By training the convolutional neural network with a U-net architecture using the simulated noise-free-noisy dataset to learn the optimal denoising function, a coarse-grained denoiser is obtained. Among them, the loss function of the convolutional neural network with a U-net architecture is: Among them, is a denoising function, is the denoising function parameter, is the image with noise and is the noise-free image, is the image dataset with noise.

5. The method according to claim 1, characterized in that, The step S14 includes performing the following operations on each image in the contrast-enhanced dataset: S141. Divide the image into an overlapping set of image patches according to a preset patch size and a preset stride; S142. Divide each image patch into a preset number of local patches to form a set of local patches, calculate the structural similarity between any two local patches in the set of local patches. Among them, the image patches with the structural similarity between any two local patches greater than a preset similarity threshold are the pure noise regions in the original image; S143. Locate the position of the pure noise region in the original image based on the coordinates of the pure noise region in the contrast-enhanced image to extract the pure noise region in the original image.

6. The method according to claim 5, characterized in that, The preset patch size is 320*320 or 640*640, the preset stride is half of the preset patch size, and the preset number is 4.

7. The method according to claim 5, characterized in that, The preset similarity threshold is 0.

7.

8. The method according to claim 1, wherein In step S2, a noise generation model is obtained by training a GAN model with a pure noise data set. The GAN model includes a generative network and a discriminative network. Among them, the generative network includes five fully connected layers, and the discriminative network includes three fully connected layers. Both the generative network and the discriminative network use the LeakyReLU activation function. The loss function of the GAN model is: Among them, is the discriminant network function, is the distribution of the pure noise dataset, is the distribution of the generative network, is the distribution uniformly sampled along a straight line between the paired points sampled from and λ is a hyperparameter of the GAN framework.

9. The method according to claim 8, wherein The hyperparameter of the GAN model is set to 10.

10. The method according to claim 1, characterized in that, In step S3, the pure noise samples generated by the noise generation model and the simulated noise-free samples in the simulated noise-free-noisy samples generated by the simulation tool are reweighted with reference to the simulated noisy samples corresponding to the simulated noise-free samples, and a new noisy sample represented by the reweighted result of the pure noise samples and the simulated noise-free samples is obtained, and a new paired noise-free-noisy sample is formed with the noise-free samples to achieve noise migration, where: The new noisy sample is expressed as: The goal of noise migration is: wherein, is a modulation function, , and are scalar coefficients, is the image block size, is a pure noise sample generated by the noise generation model, is a noise-free image generated by the simulation tool, is a noisy image corresponding to the noise-free image generated by the simulation tool.

11. The method according to claim 1, characterized in that Step S4 includes: Using the noise-free-noisy data set obtained in step S3 to train a coarse-grained denoiser to fine-tune the coarse-grained denoiser to obtain a denoising model.

12. The method according to claim 11, wherein During the process of obtaining the coarse-grained denoiser and fine-tuning to obtain the denoising model, the Adagrad optimizer is used to train the model parameters and the learning rate is 0.

001.

13. A two-dimensional cryo-electron microscopy image denoising method, characterized in that, The method includes: T1. Obtain a two-dimensional cryo-electron microscopy image to be processed; T2. Use the denoising model trained by the method according to any one of claims 1-12 to perform denoising processing on the two-dimensional cryo-electron microscopy image.

14. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 12.

15. An electronic device, characterized in that, It includes: One or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements the steps of the method according to any one of claims 1 to 12.

Images