Cryoelectron microscope cross-sectional image restoration method based on deep learning

Abstract

The invention designs a cryoelectron microscope image restoration frame IsoNet + +, belongs to the technical field of image processing and application, and aims at solving the problems that an existing cryoelectron microscope image contains a missing wedge shape and is low in signal-to-noise ratio. The method comprises the following steps: improving the signal-to-noise ratio of a cryoelectron microscope image by using a Wiener filter, and cutting the image by using a mask to provide a data basis for model iteration; the FFTUnet model and a mask are used for generating data of the round of iteration, a training pair needed by wedge-missing and noise generation model training is artificially added in the image, then the FFTUnet model is trained, the model comprises two-channel image feature extraction and a CBAM attention mechanism to pay attention to key features of the image, and the FFTUnet can better repair the cryoelectron microscope image; and finally, after the model is completely trained, inputting an unprocessed image into the model, and generating a repaired image.

Description

Technical Field

[0001] The present invention belongs to the field of biological image processing and application technology, and specifically relates to a cryo-electron microscopy tomographic image restoration method based on deep learning. Background Art

[0002] Cryo-electron tomography combines cryo-electron microscopy with computer-generated 3D reconstruction technology, enabling 3D imaging of microscopic biological samples such as cells, viruses, and proteins. This technique uses liquid nitrogen or helium to rapidly freeze biological samples to extremely low temperatures, preventing damage to the biomolecules during freezing and effectively preserving their structure for visualization in 3D images. Under extremely low temperatures, a cryo-electron microscope emits a low-dose electron beam to image the target, producing a series of 2D images. These 2D images are then combined into a 3D image using 3D reconstruction technology, enabling more detailed structural analysis of the sample.

[0003] Although cryo-electron tomography offers significant advantages for imaging microscopic biological samples, it still faces several challenges. When photographing the sample, experimental conditions limit the sample's rotation to a range of –60° to +60°, preventing full rotation from –90° to +90°. This ultimately results in information loss in the sample's three-dimensional image at angles not captured. Consequently, the corresponding three-dimensional image produces a "missing wedge" effect in Fourier space, which is reflected in real space as artifacts on the sides of the image, blurring some details. Furthermore, the low electron dose used in the imaging process and the contrast transfer function (CTF)-like effects that occur result in a low signal-to-noise ratio. The missing wedge renders the three-dimensional image incomplete, hindering a comprehensive analysis of the sample's structure, while the low signal-to-noise ratio reduces image quality and resolution. Summary of the Invention

[0004] Purpose of the Invention: The limited angles at which cryo-electron microscopes capture samples result in missing information in the reconstructed three-dimensional image. Furthermore, noise interference during the capture process blurs the two-dimensional image, resulting in a low signal-to-noise ratio (SNR) in the three-dimensional image. To address the issues of missing wedges and low SNR in three-dimensional images, the present invention designs the IsoNet++ cryo-electron microscopy image restoration framework. This framework specifically incorporates a training pair generation module and a supervised model, FFT_Unet, to improve the accuracy of three-dimensional features captured in cryo-electron microscopy images. Different loss functions are also designed to increase the model's focus on the unique characteristics of the sample.

[0005] Technical solution:

[0006] 1. The cryo-electron microscope image restoration framework IsoNet++ of the present invention mainly includes three stages: data preprocessing, model iteration and data restoration. In the data preprocessing stage, the present invention uses a Wiener filter to de-CTF convolution of the image to obtain a clear image. Convolution and filters are used at the same time to generate a mask image to locate the effective information in the cryo-electron microscope image. Finally, the deconvolved clear cryo-electron microscope image is divided into a large number of small-sized images (partial images) according to the mask image, which are used as samples for FFT_Unet model training. In the model training stage, partial images are used to construct the training pairs required for model training, and these training pairs are used to train the model. In the data restoration stage, the deconvolved image or the original image is input into the model that finally completes the training for restoration. Specifically, the following steps:

[0007] (1) In the data preprocessing stage, the present invention uses a Wiener filter to deconvolve the original cryo-EM image to solve the interference of the contrast transfer function (CTF) during the imaging process and improve the image clarity. Most areas in the cryo-EM image are invalid areas composed of air and water. In order to improve the quality of the training data set, these invalid areas need to be avoided. The present invention combines the convolution operation with the filter to generate a three-dimensional mask image for accurately locating the effective information area in the image. Based on the three-dimensional mask image, the clear image after convolution processing is divided into multiple partial images, providing high-quality input data for subsequent model training to improve the model's image repair ability.

[0008] (2) The model iteration stage includes a data iteration module, a training pair generation module, and a repair model training module. In the data iteration module, the present invention first inputs a partial image into the preliminarily trained FFT_Unet model to obtain a repaired image (corrected image), and then combines the partial image and the corrected image in Fourier space according to the shape of the mask to obtain a combined image. In the training pair generation module, the present invention adds missing wedges and noise to the combined image to obtain an image with more missing wedges and noise as the input image, which constitutes a training pair of the FFT_Unet model with the combined image (target image). In the repair model training module, the present invention uses the training pair to train the FFT_Unet model, and the trained FFT_Unet model provides a model basis for the next data iteration.

[0009] (3) In the data restoration phase, the cryo-EM images are preprocessed using filters, then cut into smaller images using masks. These smaller images are then fed into the FFT_Unet model for restoration. The restored images are then reassembled to obtain a complete, high-quality cryo-EM image.

[0010] The data preprocessing operation is:

[0011] The method of the present invention requires an input image size of H×W×C. H, W, and C are positive integers, and the original image size is H×W×C. The cryo-electron microscopy image is first convolved and then cut into a large number of partial images of size 96×96×96.

[0012] The data iteration module is:

[0013] The present invention inputs a large number of partial images into a preliminarily trained FFT_Unet model to obtain corrected images. The present invention then transforms the partial and corrected images into Fourier space and combines them according to the shape of the mask. The images are then inverse Fourier transformed into real space to obtain the combined image required for this round of model training.

[0014] The training pair generation module is:

[0015] After obtaining a large number of combined images (target images), the present invention first rotates and transforms the target images into Fourier space, then adds a mask to simulate the missing wedges inherent in the image. The masked images are then inverse Fourier transformed back into real space, yielding cryo-EM images containing artificial missing wedges. Finally, Gaussian noise of varying intensities is added to the images, depending on the number of iterations, to obtain input images with more missing wedges and noise. After these processes, the target and input images are obtained, which provide the input-output training pairs for training the FFT_Unet model.

[0016] The data repair operation is as follows:

[0017] This method cuts the original cryo-EM image (or the image after CTF deconvolution) into small-size images (96 x 96 x 96). These small-size images are then fed into the trained FFT_Unet model for restoration. The restored small-size images are then reassembled to obtain a complete, high-quality cryo-EM image.

[0018] 2. The cryo-electron microscopy image restoration method according to claim 1, wherein the loss function includes a Mask Loss loss function and a SSIM Loss loss function, and then the two types of loss functions are used to train models separately to restore the features of the cryo-electron microscopy image, and the Mask loss and SSIM loss minimize the following objective function:

[0019]

[0020] in, Indicates taking a mask for the cryo-electron microscopy image. represents the corrected image in Fourier space, Represents the target image in Fourier space.

[0021]

[0022] in, Represents two images, express The average value of express The average value of express The variance of express The variance of express and The covariance of , is a constant, is the dynamic range of pixel values.

[0023] 3. The repair method according to claim 1 is characterized in that the learning iteration cycle is 30 cycles, the number of training rounds in each cycle is 200, and in order to enable IsoNet++ to have the ability to remove noise, the present invention adds different Gaussian noises in different iteration cycles. No noise is added to the data in the first 10 iterations, and the noise intensity is increased by 0.05 every 5 iterations after 10 iterations. The learning rate used in the FFT_Unet model is 1e-5.

[0024] Beneficial effects:

[0025] The IsoNet++ cryo-EM restoration framework was trained using multiple datasets to ensure the authority of the training.

[0026] Use filters and convolution modules to process raw cryo-EM images to improve the signal-to-noise ratio of the images. Use masks to segment the valid areas in the images to prepare data for model training.

[0027] We use a dual-channel feature extraction network and utilize the CBAM attention mechanism to extract features from cryo-EM images. During data iteration, we combine model-repaired cryo-EM images with unprocessed images as pre-training data to continuously improve the model's repair capabilities.

[0028] Mask loss and SSIM loss are used as loss functions in the deep learning FFT_Unet model to improve the effect of IsoNet++ framework in repairing cryo-electron microscopy images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is the overall framework diagram of IsoNet++;

[0030] Figure 2This is the FFT_Unet model diagram;

[0031] Figure 3 This is the CBAM structure diagram;

[0032] Figure 4 Add missing wedges and noise flow chart for artificial;

[0033] Figure 5 Add missing wedges and noise effects to the data;

[0034] Figure 6 Schematic diagram of the cryo-electron microscopy image restoration results. DETAILED DESCRIPTION

[0035] Example: The IsoNet++ cryo-EM image restoration framework provided by the present invention is used to repair missing wedges and low signal-to-noise ratio in cryo-EM images. The specific operation is performed as follows:

[0036] 1. Cryo-EM datasets

[0037] The cryo-electron microscopy (Cryo-EM) data used in this invention are sourced from multiple platforms. The HIV dataset is from the Figshare platform, the neuron synapse dataset is from the Figshare platform, the EMPIAR-11078 dataset is from the EMPIAR platform, and the SHREC2021 dataset is obtained through the DateverseNL platform. The HIV dataset contains three cryo-EM images, and the neuron synapse dataset contains one cryo-EM image. The EMPIAR-11078 dataset contains 20 cryo-EM images. This paper selects the first three images from the SHREC 2021 dataset and, through artificial synthesis, adds Gaussian noise of three intensity levels (0.5, 0.25, and 0.125) to each image. The data with Gaussian noise of the same intensity and missing wedges are then superimposed, resulting in a total of six types of synthetic data.

[0038] 2. Data Preprocessing

[0039] The method of the present invention requires that the large-scale cryo-electron microscopy image is first subjected to filtering and convolution and then segmented into 96x96x96 images. During model training, the 96x96x96 data is then resized to 64x64x64.

[0040] 3. Model Iteration

[0041] First, a partial image is passed through a pre-trained model to generate a corrected image. A mask is then used to create a composite image from the corrected and partial images. Then, the composite image (the target image) is artificially added with missing wedges and noise to generate the input image. Finally, the input and target images form the training pair required for the model.

[0042] During network training, data is input in batches. Preprocessing begins by converting 96×96×96 images to 64×64×64 images. These 64×64×64 images are then fed into the FFT_Unet model. The FFT_Unet model consists of an encoder module and a decoder module. In the encoder module, the input data is first processed in two channels. The left channel data is left untouched, and then a downsampling module is used to extract features. In the right channel, the data is first converted to Fourier space, and then a downsampling module is used to extract features. In the decoder module, the downsampled data from the two channels is added together. The CBAM module is then used to extract important features from the cryo-EM image. These extracted features are then upsampled. Finally, a series of upsampling steps yield the restored cryo-EM image.

[0043] The main function of the convolutional block in upsampling is to increase the number of channels and size. This paper adds a CBAM module to the upsampling process. The CBAM module blends information from different channels with spatial information, extracting important features and suppressing unnecessary ones, thereby making the restored image more complete and accurate. Within the CBAM module, the channel attention module captures the maximum and average values ​​in three-dimensional data, while the spatial attention module captures the maximum and average values ​​in two-dimensional data. Therefore, after passing the channel attention module and the spatial attention module, the data can fuse the channel and spatial information, allowing the model to learn the important features of cryo-EM images during training, resulting in more accurate restoration of cryo-EM images.

[0044] Finally, the original cryo-EM image is cut into smaller images and fed into the trained FFT_Unet model for restoration. The restored smaller images are then reassembled to obtain a complete, high-quality cryo-EM image.

[0045] The loss function is calculated for the input image with more missing wedges and noise added and the target image, and the network parameters are updated. The Mask loss function and SSIM loss function of the present invention are used to train the FFT_Unet model respectively, minimizing the following objective function:

[0046]

[0047] in, Indicates taking a mask for the cryo-electron microscopy image. represents the corrected image in Fourier space, Represents the target image in Fourier space. The smaller the pixel difference between the restored cryo-EM image after masking and the target cryo-EM image, the smaller the loss. The objective function is as follows:

[0048]

[0049] in, Represents two images, express The average value of express The average value of express The variance of express The variance of express and The covariance of , is a constant, is the dynamic range of pixel values. The smaller the structural difference between the restored cryo-EM image and the target cryo-EM image, the smaller the loss.

[0050] 4. Analysis of processing results

[0051] This method uses the following three performance metrics to quantify the processing results: normalized correlation coefficient (CC), structural similarity (SSIM), and peak signal-to-noise ratio (PSNR). The calculation formula is as follows:

[0052]

[0053] in, represents the restored image, represents the target image, Indicates taking the average value of the image.

[0054]

[0055] in, Represents two images, express The average value of express The average value of express The variance of express The variance of express and The covariance of , is a constant, is the dynamic range of pixel values.

[0056]

[0057] in, , is the maximum value representing the image, Indicates the width of the image. Indicates the height of the image, represents the channels of the image, represents the restored image, Represents the target image.

[0058] The proposed method achieved CC of 0.93, SSIM of 0.06, and PSNR of 17 on a synthetic dataset with 0.125-intensity Gaussian noise. Extensive application demonstrates that the proposed IsoNet++ cryo-EM image restoration framework achieves excellent restoration results. This is of great significance in the field of biological imaging.

[0059] As described above, although the present invention has been shown and described with reference to certain preferred embodiments, it is not to be construed as limiting the invention itself. Various changes in form and details may be made thereto without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. The cryo-electron microscope image restoration framework IsoNet++ of the present invention mainly includes three stages: data preprocessing, model iteration and data restoration. In the data preprocessing stage, the present invention uses a Wiener filter to de-CTF convolution of the image to obtain a clear image. Convolution and filters are used at the same time to generate a mask image to locate the effective information in the cryo-electron microscope image. Finally, the deconvolved clear cryo-electron microscope image is divided into a large number of small-sized images (partial images) according to the mask image as samples for FFT_Unet model training. In the model training stage, partial images are used to construct the training pairs required for model training, and these training pairs are used to train the model. In the data restoration stage, the deconvolved image or the original image is input into the model that finally completes the training for restoration. Specifically, the following steps: (1) In the data preprocessing stage, the present invention uses a Wiener filter to perform CTF deconvolution on the original cryo-EM image to solve the interference of similar contrast transfer function (CTF) during the imaging process and improve the image clarity. Most areas in the cryo-EM image are invalid areas composed of air and water. In order to improve the quality of the training data set, these invalid areas need to be avoided. The present invention combines the convolution operation with the filter to generate a three-dimensional mask image for accurately locating the effective information area in the image. According to the three-dimensional mask image, the clear image after convolution processing is divided into multiple partial images, providing high-quality input data for subsequent model training to improve the model's ability to repair the image; (2) The model iteration stage includes a data iteration module, a training pair generation module, and a repair model training module. In the data iteration module, the present invention first inputs a partial image into the preliminarily trained FFT_Unet model to obtain a repaired image (corrected image), and then combines the partial image and the corrected image in Fourier space according to the shape of the mask to obtain a combined image. In the training pair generation module, the present invention adds missing wedges and noise to the combined image to obtain an image with more missing wedges and noise as the input image, which constitutes a training pair of the FFT_Unet model with the combined image (target image). In the repair model training module, the present invention uses the training pair to train the FFT_Unet model, and the trained FFT_Unet model provides a model basis for the next data iteration; (3) In the data restoration stage, the cryo-EM image is preprocessed using a filter, and then the cryo-EM image is cut into small-sized images using a mask. These small-sized images are then input into the FFT_Unet model for restoration. The restored images are then reassembled to obtain a complete high-quality cryo-EM image. The data preprocessing operation is: The method of the present invention requires an input image size of H×W×C. H, W, and C are positive integers, and the original image size is H×W×C. The cryo-electron microscopy image is first convolved and then cut into a large number of partial images of size 96×96×96.

2. The data iteration module is: The present invention inputs a large number of partial images into a pre-trained FFT_Unet model to obtain corrected images. The present invention then transforms the partial and corrected images into Fourier space and combines them according to the shape of the mask. The images are then inverse Fourier transformed into real space to obtain the combined image required for this round of model training. The training pair generation module is: After obtaining a large number of combined images (target images), the present invention first rotates and converts the target image into Fourier space, and then adds a mask to simulate the missing wedges in the image itself. The masked image is then inverse Fourier transformed back to real space to obtain a cryo-EM image containing artificial missing wedges. Finally, Gaussian noise of varying intensities is added to the image according to the number of iterations to obtain an input image with more missing wedges and noise. After the above processing, the target image and the input image are obtained, which provide input and output training pairs for training the FFT_Unet model. The data repair operation is as follows: The present invention cuts the original cryo-EM image (or the image after CTF deconvolution) into small-size images (96 x 96 x 96) and inputs these small-size images into the finally trained FFT_Unet model for repair. The repaired small-size images are reassembled to obtain a complete high-quality cryo-EM image. The cryo-electron microscopy image restoration method according to claim 1 is characterized in that: The loss functions include MaskLoss loss function and SSIM Loss loss function. The two types of loss functions are then used to train the models separately to repair the features of the cryo-EM image. Mask loss and SSIM loss minimize the following objective function: in, Indicates taking a mask for the cryo-electron microscopy image. represents the corrected image in Fourier space, Represent the target image in Fourier space; in, Represents two images, express The average value of express The average value of express The variance of express The variance of express and The covariance of , is a constant, is the dynamic range of pixel values.

3. The repair method according to claim 1, wherein: The learning iteration cycle is 30 cycles, and the number of training rounds in each cycle is 200. In order to enable IsoNet++ to have the ability to remove noise, the present invention adds different Gaussian noises in different iteration cycles. No noise is added to the data in the first 10 iterations. After 10 iterations, the noise intensity is increased by 0.05 every 5 times. The learning rate used by the FFT_Unet model is 1e-5.

Images