Methods, apparatus and equipment for training reconstruction models of three-dimensional electron microscopy images

Abstract

This application discloses a method, apparatus, and device for training a reconstruction model of three-dimensional electron microscope images, belonging to the field of artificial intelligence technology. The method includes: acquiring missing electron microscope image samples, image mask samples, and complete electron microscope image samples in three-dimensional image format; inputting the masked electron microscope image obtained by multiplying the missing electron microscope image samples and the image mask samples into the reconstruction model for k-fold iterative reconstruction to obtain a reconstructed electron microscope image; and training the model parameters of the reconstruction model based on the error between the complete electron microscope image samples and the reconstructed electron microscope image. The three-dimensional electron microscope image reconstruction model trained by the above method can effectively recover information lost in cryo-electron tomography data with low signal-to-noise ratios.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and equipment for training a reconstruction model of a three-dimensional electron microscope image. Background Technology

[0002] 3D reconstruction refers to the technique of using a series of 2D projected images along different directions and a reconstruction algorithm to reconstruct the 2D projected images into 3D images, thereby obtaining a 3D image.

[0003] In related technologies, iterative compressed sensing-optimized non-uniform fast Fourier transform is typically used for 3D reconstruction of MRI data. By combining compressed sensing and non-uniform Fourier transform, the natural sparsity of the sample and the prior knowledge that its density is non-negative compared to non-solvents are utilized to recover information lost during the imaging process.

[0004] Because MRI data has a very high signal-to-noise ratio, and the missing information is mainly in the high-frequency region of Fourier space, related techniques can recover the lost information well in the three-dimensional reconstruction of MRI data. However, in the reconstruction of cryo-electron microscopy (cryo-EM) tomography data, due to the lower signal-to-noise ratio and the fact that the missing information is mainly caused by the limited acquisition angle, related techniques are less effective at recovering the lost information in cryo-EM tomography data. Summary of the Invention

[0005] This application provides a method, apparatus, and device for training a reconstruction model of three-dimensional electron microscope images, which can effectively recover lost information from cryo-electron tomography data. The technical solution is as follows:

[0006] According to one aspect of this application, a method for training a reconstruction model of a three-dimensional electron microscope image is provided, the method comprising:

[0007] Acquire missing electron microscope image samples, image mask samples, and complete electron microscope image samples in a three-dimensional image format. The missing electron microscope image samples are cryo-electron microscope images with missing data. The image mask samples are used to mask invalid pixels in the missing electron microscope image samples. The complete electron microscope image samples are cryo-electron microscope images without missing data.

[0008] The masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample is input into the reconstruction model for k iterations of reconstruction to obtain the reconstructed electron microscope image, where k is an integer greater than 0.

[0009] The model parameters of the reconstruction model are trained based on the error between the complete electron microscope image sample and the reconstructed electron microscope image.

[0010] According to another aspect of this application, a method for reconstructing a three-dimensional electron microscope image is provided, the method comprising:

[0011] Acquire missing electron microscopy images and image masks in a three-dimensional image format, wherein the missing electron microscopy images are cryo-electron microscopy images with missing data;

[0012] The masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask is input into the reconstruction model for k iterations of reconstruction to obtain the reconstructed electron microscope image, where k is an integer greater than 0;

[0013] Output the reconstructed electron microscope image.

[0014] According to another aspect of this application, a training apparatus for a reconstruction model of three-dimensional electron microscopy images is provided, the apparatus comprising:

[0015] The sample acquisition module is used to acquire missing electron microscope image samples, image mask samples, and complete electron microscope image samples in a three-dimensional image format. The missing electron microscope image samples are cryo-electron microscope images with missing data. The image mask samples are used to mask invalid pixels in the missing electron microscope image samples. The complete electron microscope image samples are cryo-electron microscope images without missing data.

[0016] The sample iteration module is used to input the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into the reconstruction model for k loop iterations to reconstruct the reconstructed electron microscope image.

[0017] The training module is used to train the model parameters of the reconstruction model based on the error between the complete electron microscope image sample and the reconstructed electron microscope image.

[0018] According to another aspect of this application, a three-dimensional electron microscope image reconstruction apparatus is provided, the apparatus comprising:

[0019] The acquisition module is used to acquire a missing electron microscope image in a three-dimensional image format and the image mask, wherein the missing electron microscope image is a cryo-electron microscope image with missing data;

[0020] An iterative module is used to input the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask into the reconstruction model for k iterative reconstructions to obtain the reconstructed electron microscope image.

[0021] The output module is used to output the reconstructed electron microscope image.

[0022] According to another aspect of this application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the three-dimensional electron microscope image reconstruction model training method as described above, or the three-dimensional electron microscope image reconstruction method as described above.

[0023] According to another aspect of this application, a computer storage medium is provided, wherein at least one computer instruction is stored in the computer-readable storage medium, the at least one computer instruction being loaded and executed by a processor to implement the three-dimensional electron microscope image reconstruction model training method as described above, or the three-dimensional electron microscope image reconstruction method as described above.

[0024] According to another aspect of this application, a computer program product is provided, the computer program product comprising computer instructions stored in a computer-readable storage medium; the computer instructions are read from and executed by a processor of a computer device from the computer-readable storage medium, causing the computer device to perform the three-dimensional electron microscope image reconstruction model training method as described above, or the three-dimensional electron microscope image reconstruction method as described above.

[0025] The beneficial effects of the technical solution provided in this application include at least the following:

[0026] In this embodiment, a masked electron microscope image, obtained by multiplying a missing electron microscope image sample and an image mask sample, is input into a reconstruction model for k iterations to obtain a reconstructed electron microscope image. Based on the error between the complete electron microscope image sample and the reconstructed electron microscope image, the model parameters of the reconstruction model are trained to finally obtain a reconstruction model of the three-dimensional electron microscope image. The reconstruction model of the three-dimensional electron microscope image obtained through training can recover the information lost in cryo-electron tomography data with low signal-to-noise ratio relatively well. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of a computer system for training a reconstruction model of a three-dimensional electron microscope image provided in an exemplary embodiment of this application;

[0029] Figure 2 This is a flowchart of a method for training a reconstruction model of a three-dimensional electron microscope image provided in an exemplary embodiment of this application;

[0030] Figure 3 This is a flowchart of a method for reconstructing a three-dimensional electron microscope image provided in an exemplary embodiment of this application;

[0031] Figure 4 This is a framework diagram of the generation of a reconstruction model for a three-dimensional electron microscope image and the reconstruction of a three-dimensional electron microscope image provided in an exemplary embodiment of this application;

[0032] Figure 5 This is a schematic diagram of the structure of a training framework for a reconstruction model of a three-dimensional electron microscope image provided in an exemplary embodiment of this application;

[0033] Figure 6 This is a schematic diagram of the structure of a noise reduction network provided in an exemplary embodiment of this application;

[0034] Figure 7 This is a flowchart of a method for training a reconstruction model of a three-dimensional electron microscope image provided in an exemplary embodiment of this application;

[0035] Figure 8 This is a flowchart of a method for reconstructing a three-dimensional electron microscope image provided in an exemplary embodiment of this application;

[0036] Figure 9 This is a structural block diagram of a three-dimensional electron microscope image reconstruction model training device provided in an exemplary embodiment of this application;

[0037] Figure 10 This is a structural block diagram of a three-dimensional electron microscope image reconstruction apparatus provided in an exemplary embodiment of this application;

[0038] Figure 11 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application.

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0041] Figure 1 A schematic diagram of a computer system provided in one embodiment of this application is shown. This computer system can implement a system architecture for a method of training a reconstruction model of a three-dimensional electron microscope image. The computer system may include: a terminal 100 and a server 200.

[0042] Terminal 100 can be an electronic device such as a mobile phone, tablet computer, vehicle terminal (vehicle system), wearable device, PC (Personal Computer), access control device, or unmanned vending terminal. A client application for the target application can be installed and run on terminal 100. This target application can be a game application or other application that provides training functions for reconstructed models based on 3D electron microscope images; this application does not limit the specific form of the target application. Furthermore, this application does not limit the form of the target application, including but not limited to apps, mini-programs, etc., installed on terminal 100, and can also be in web page form.

[0043] Server 200 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Server 200 can be the backend server for the aforementioned target application, used to provide backend services to the clients of the target application.

[0044] Terminal 100 and server 200 can communicate via a network, such as a wired or wireless network.

[0045] The method for training a reconstruction model of three-dimensional electron microscope images provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1 Taking the implementation environment of the scheme shown as an example, the reconstruction model training method of the three-dimensional electron microscope image can be executed by the terminal 100 (such as the client of the target application installed and running in the terminal 100 executing the reconstruction model training method of the three-dimensional electron microscope image), or the reconstruction model training method of the three-dimensional electron microscope image can be executed by the server 200, or the terminal 100 and the server 200 can interact and cooperate to execute it. This application does not limit this.

[0046] Furthermore, the technical solution of this application can be combined with blockchain technology. For example, in the method for training a reconstruction model of three-dimensional electron microscope images disclosed in this application, some of the data involved (such as three-dimensional images, reconstructed images, etc.) can be stored on the blockchain.

[0047] For ease of explanation, in the following method embodiments, only computer devices will be used as the execution subject for each step of the method for training a reconstruction model of three-dimensional electron microscopy images.

[0048] This application's technical solution mainly targets the 3D reconstruction of cryo-electron microscopy (cryo-EM) tomography data. Cryo-EM tomography data has a lower signal-to-noise ratio, making reconstruction difficult. When the sample is thick, the characteristics of electron scattering cannot effectively reconstruct the 3D structure of the cryo-EM tomography data. Furthermore, when there are many background impurities in the sample, the 3D reconstruction process is easily affected by these impurities. Therefore, this application proposes a method for 3D reconstruction of cryo-EM tomography data based on an AI model. This method involves inputting a masked electron microscope image obtained by multiplying a missing electron microscope image sample and an image mask sample into a reconstruction model for k iterations to obtain a reconstructed electron microscope image. Based on the error between the complete electron microscope image sample and the reconstructed electron microscope image, the model parameters of the reconstruction model are trained, ultimately yielding a 3D electron microscope image reconstruction model.

[0049] Figure 2 This application illustrates a flowchart of a method for training a reconstruction model of a three-dimensional electron microscope image provided in an exemplary embodiment. This method can be executed by a computing device, such as... Figure 2 As shown, the method for training a reconstruction model of a 3D electron microscope image may include the following steps:

[0050] Step 202: Obtain missing electron microscopy image samples, image mask samples, and complete electron microscopy image samples in three-dimensional image format.

[0051] Computer equipment acquires missing electron microscope image samples, image mask samples, and complete electron microscope image samples in three-dimensional image format.

[0052] Missing electron microscopy image samples are cryo-electron microscopy images with missing data. For example, when cryo-electron tomography (Cryo-ET) scans a sample, the scanning angle cannot cover the full 180 degrees, such as ±45° or ±60°. The information obtained at this scanning angle is incomplete. The missing scanning angle makes the structure of the cryo-electron microscopy image incomplete and also easily leads to the distortion of the cryo-electron microscopy image.

[0053] An image mask template is used to mask invalid pixels in a missing electron microscopy image sample. For example, the image mask is used to describe data loss in a cryo-electron microscopy image; or, the image mask is used to describe data loss caused by the limited scanning angle range when performing tomographic scanning of a sample using cryo-electron tomography.

[0054] A complete electron microscopy image sample refers to a cryo-electron microscopy image without any missing data.

[0055] Step 204: Input the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into the reconstruction model and perform k iterations of reconstruction to obtain the reconstructed electron microscope image.

[0056] Given missing electron microscope image samples and image mask samples, the missing electron microscope image samples and image mask samples are multiplied together, and the resulting masked electron microscope image is input into the reconstruction model for k iterations to finally obtain the reconstructed electron microscope image, where k is an integer greater than 0.

[0057] In one possible implementation, the reconstruction model is a recurrent neural network. The computer device inputs the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into the recurrent neural network for k iterations of reconstruction, and finally obtains the reconstructed electron microscope image, where k is an integer greater than 0.

[0058] Step 206: Train the model parameters of the reconstruction model based on the error between the complete electron microscope image sample and the reconstructed electron microscope image.

[0059] The computer equipment trains the model parameters of the reconstruction model based on the error between the complete electron microscope image sample and the reconstructed electron microscope image. The reconstructed model after training is used to process the input missing electron microscope image and image mask.

[0060] In summary, the solution presented in this application involves acquiring missing electron microscope image samples, image mask samples, and complete electron microscope image samples in a three-dimensional image format. A computer device multiplies the missing electron microscope image samples and the image mask samples to obtain a masked electron microscope image, which is then input into a reconstruction model for k iterations to obtain a reconstructed electron microscope image. Based on the error between the complete electron microscope image samples and the reconstructed electron microscope image, the computer device trains the model parameters of the reconstruction model to obtain a trained reconstruction model. The three-dimensional electron microscope image reconstruction model training method provided in this application, through the reconstruction model obtained through k iterations, can effectively recover information lost in cryo-electron tomography data with low signal-to-noise ratios.

[0061] Based on the above Figure 2 After the reconstruction model is trained using the scheme shown in the embodiment, it can be applied to various reconstruction scenarios of 3D electron microscope images. The reconstruction process of the reconstruction model for 3D electron microscope images can be referred to in the following embodiment.

[0062] Figure 3This is a flowchart illustrating a method for reconstructing a three-dimensional electron microscope image according to an exemplary embodiment. The method can be executed by a computer device, such as a server, a terminal, or a combination of both. The server can be one of the aforementioned... Figure 1 The server 200 in the illustrated embodiment can be the terminal described above. Figure 1 The terminal 100 in the illustrated embodiment. This computer device can be implemented as an application device for reconstructing models from three-dimensional electron microscope images. For example... Figure 3 As shown, the method for reconstructing the three-dimensional electron microscope image may include the following steps.

[0063] Step 302: Obtain the missing electron microscope image and image mask in a three-dimensional image format.

[0064] Missing electron microscopy image samples are cryo-electron microscopy images with missing data; image mask samples are used to mask invalid pixels in missing electron microscopy image samples.

[0065] Step 304: The masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask is input into the reconstruction model for k iterations of reconstruction to obtain the reconstructed electron microscope image.

[0066] The reconstruction model is a reconstruction model trained according to steps 202-206 in the above embodiments.

[0067] Step 306: Output the reconstructed electron microscope image.

[0068] In summary, the solution presented in this application involves acquiring a missing electron microscope image and an image mask in a three-dimensional image format; multiplying the missing electron microscope image and the image mask to obtain a masked electron microscope image, which is then input into a reconstruction model for k iterations of reconstruction to obtain and output a reconstructed electron microscope image. The three-dimensional electron microscope image reconstruction method provided in this application, through a trained reconstruction model, can effectively recover information lost in cryo-electron tomography data with low signal-to-noise ratios.

[0069] The solution described in the above embodiments of this application can be implemented by calling a pre-trained reconstruction model. In other words, the solution involved in this application includes a three-dimensional electron microscope image reconstruction model generation stage and a three-dimensional electron microscope image reconstruction stage. Figure 4 This is a framework diagram illustrating the generation of a reconstruction model for a three-dimensional electron microscope image and the reconstruction of the three-dimensional electron microscope image according to an exemplary embodiment, such as... Figure 4As shown, in the three-dimensional electron microscope image reconstruction model generation stage, the three-dimensional electron microscope image reconstruction model generation device 410 trains the three-dimensional electron microscope image reconstruction model using a pre-set training sample dataset (including different missing electron microscope image samples, image mask samples, and complete electron microscope image samples corresponding to the missing electron microscope image samples), thereby obtaining a trained three-dimensional electron microscope image reconstruction model. In the three-dimensional electron microscope image reconstruction stage, the three-dimensional electron microscope image reconstruction device 420 processes the input missing electron microscope image based on the three-dimensional electron microscope image reconstruction model to obtain the reconstructed electron microscope image corresponding to the missing electron microscope image.

[0070] The aforementioned three-dimensional electron microscope image reconstruction model generation device 410 and three-dimensional electron microscope image reconstruction device 420 can be computer devices. For example, the computer device can be a fixed computer device such as a personal computer or a server, or it can be a mobile computer device such as a tablet computer or an e-book reader.

[0071] Optionally, the aforementioned 3D electron microscope image reconstruction model generation device 410 and 3D electron microscope image reconstruction device 420 can be the same device, or they can be different devices. Furthermore, when the 3D electron microscope image reconstruction model generation device 410 and 420 are different devices, they can be of the same type, such as both being servers; or they can be of different types, such as 420 being a personal computer or terminal, while 410 can be a server, etc. The specific types of the three-dimensional electron microscope image reconstruction model generation device 410 and the three-dimensional electron microscope image reconstruction device 420 are not limited in the embodiments of this application.

[0072] Figure 5 A schematic diagram of the structure of a training framework for a reconstruction model of a three-dimensional electron microscope image provided in an exemplary embodiment of this application is shown. The reconstruction model of the three-dimensional electron microscope image includes a recurrent neural network, which includes a denoising network 504 and a data consistency network 505. Based on the structure of this reconstruction model of the three-dimensional electron microscope image, the following method embodiment is proposed.

[0073] The embodiments of this application involve a missing electron microscope image sample 501, an image mask sample 502, a complete electron microscope image sample 503, a noise reduction network 504, a data consistency network 505, and a weight sharing module 506.

[0074] The computer equipment acquires a missing electron microscope image sample 501, an image mask sample 502, and a complete electron microscope image sample 503 in a three-dimensional image format.

[0075] An image mask template is used to mask invalid pixels in a missing electron microscope image sample. For example, the image mask is used to describe data loss in an electron microscope image; or, the image mask is used to describe data loss caused by the limited scanning angle range during cryo-electron tomography (cryo-electron tomography) of a sample. Viewed from the side, the image mask has an X-shape. The shaded portion 502a of the X-shaped image mask describes the data loss area caused by the limited scanning angle range during cryo-electron tomography of the sample, while the remaining portion 502b of the X-shaped image mask describes the area that can be normally scanned during cryo-electron tomography of the sample.

[0076] For example, if the tomographic scanning coverage angle of cryo-electron tomography is ±45°, then the shaded portion 502a in the X-shaped image mask is used to represent the data missing areas +45° to +125° and -45° to -125° caused by the limited scanning angle range; the remaining portion 502b in the X-shaped image mask is used to describe the areas -45° to +45°, +125° to +180° and -125° to -180° that can be normally scanned by cryo-electron tomography when scanning the sample.

[0077] The computer device inputs the missing electron microscope image sample 501 and the image mask sample 502 into the recurrent neural network for k iterations of reconstruction. The recurrent neural network includes a noise reduction network 504 and a data consistency network 505.

[0078] The computer device inputs the missing electron microscope image sample 501 and the image mask sample 502 into the recurrent neural network for the first iteration. During the first iteration, the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample is input into the denoising network 504. The denoising network 504 outputs the first reconstructed feature map Z0. The computer device inputs the masked electron microscope image and the first reconstructed feature map Z0 into the data consistency network 505 for processing. The data consistency network 505 outputs the first reconstructed image X1.

[0079] The computer device inputs the missing electron microscope image sample 501 and image mask sample 502 into the recurrent neural network for the i-th iteration. During the i-th iteration, the computer device reconstructs the i-1-th image X. i-1 The input is fed into the denoising network 504 for denoising processing, and the denoising network 504 outputs the i-th reconstructed feature map Z. iThe computer equipment will combine the masked electron microscope image and the i-th reconstructed feature map Z. i The data is input to the data consistency network 505, and the data consistency network 505 outputs the i-th reconstructed image X. i .

[0080] The computer device inputs the masked electron microscope image obtained by multiplying the missing electron microscope image sample 501 and the image mask sample 502 into the recurrent neural network for the k-th iteration. During the k-th iteration, the computer device reconstructs the image X of the (k-1)-th generation. k-1 The input is fed into the denoising network 504 for denoising processing, and the output of the denoising network 504 is the k-th reconstructed feature map Z. k The computer equipment will display the masked electron microscope image and the k-th reconstructed feature map Z. k The data is input to the data consistency network 505, and the data consistency network 505 outputs the reconstructed image X. k .

[0081] Based on reconstructed image X k Using complete electron microscope image samples 503, the mean square error of the reconstruction model is calculated. The computer device inputs the mean square error to the weight sharing module 506, and the weight sharing module 506 updates the parameters of the denoiser network 504 in the recurrent neural network based on the mean square error.

[0082] After updating the parameters of the denoising network 504 in the recurrent neural network, the next missing electron microscope image sample 501 can be selected, and the above steps can be repeated to continue training.

[0083] Figure 6 A schematic diagram of the structure of a noise reduction network provided in an exemplary embodiment of this application is shown. The noise reduction network includes a three-dimensional convolutional kernel layer 601, a batch normalization layer 602, and a ReLU activation layer 603.

[0084] The noise reduction network involved in the embodiments of this application includes n three-dimensional convolutional layers, where n is an integer greater than 1. Among the n three-dimensional convolutional layers, the 1st to n-1th three-dimensional convolutional layers include a three-dimensional convolutional kernel layer 601, a batch normalization layer 602, and a ReLU activation layer 603; the nth three-dimensional convolutional layer includes a three-dimensional convolutional kernel layer 601 and a batch normalization layer 602.

[0085] In the i-th iteration, the computer device inputs the masked electron microscope image obtained by multiplying the (i-1)-th reconstructed image and the image mask sample into the denoising network, which includes n three-dimensional convolutional layers. In the j-th convolution process in the denoising network, the computer device inputs the (j-1)-th convolution result into the j-th three-dimensional convolutional layer of the n three-dimensional convolutional layers for convolution. If j is greater than 1 and less than n, the (j-1)-th convolution result is input into the three-dimensional convolution kernel layer 601 of the j-th three-dimensional convolutional layer of the n three-dimensional convolutional layers for convolution. The three-dimensional convolution kernel layer 601 of the j-th three-dimensional convolutional layer outputs the j-th convolutional feature.

[0086] The computer device inputs the j-th convolutional feature into the batch normalization layer 602 of the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for normalization processing, and the batch normalization layer 802 outputs the normalized j-th convolutional feature.

[0087] The computer device inputs the normalized j-th convolutional feature into the ReLU activation layer 603 in the j-th three-dimensional convolutional layer among n three-dimensional convolutional layers for activation processing, and the ReLU activation layer 803 outputs the activated j-th convolutional feature.

[0088] In the j-th convolution process in the noise reduction network, the computer device inputs the (j-1)-th convolution result into the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution. When j equals n, the (j-1)-th convolution result is input into the n-th three-dimensional convolutional kernel layer 601 among the n three-dimensional convolutional layers for convolution. The three-dimensional convolutional kernel layer 601 outputs the n-th convolutional feature.

[0089] The nth convolutional feature is input into the batch normalization layer 602 in the nth three-dimensional convolutional layer to perform normalization processing, and the normalized nth convolutional feature is output as the i-th reconstructed feature map.

[0090] During the (i+1)th iteration, the above steps are repeated to continue the noise reduction process.

[0091] Figure 7 A flowchart illustrating a method for training a reconstruction model of a three-dimensional electron microscope image provided in an exemplary embodiment of this application is shown. This method can be executed by a computing device, which can be implemented as follows: Figure 4 The device 410 for generating a reconstruction model of the three-dimensional electron microscope image shown is as follows: Figure 7 As shown, the method for training the reconstruction model of the three-dimensional electron microscope image may include the following steps:

[0092] Step 702: Obtain missing electron microscopy image samples, image mask samples, and complete electron microscopy image samples in three-dimensional image format.

[0093] Computer equipment acquires missing electron microscope image samples, image mask samples, and complete electron microscope image samples in three-dimensional image format.

[0094] Missing electron microscopy image samples are cryo-electron microscopy images with missing data. For example, when cryo-electron tomography (Cryo-ET) scans a sample, the scanning angle cannot cover the full 180 degrees, such as ±45° or ±60°. The information obtained at this scanning angle is incomplete. The missing scanning angle makes the structure of the electron microscopy image incomplete and also easily leads to the distortion of the electron microscopy image.

[0095] An image mask template is used to mask invalid pixels in a missing electron microscope image sample. For example, the image mask is used to describe data loss in an electron microscope image; or, the image mask is used to describe data loss caused by the limited scanning angle range when performing tomographic scanning of a sample using cryo-electron tomography. Viewed from the side, the image mask has an X-shape.

[0096] Complete electron microscopy image samples are cryo-electron microscopy images without missing data. Complete electron microscopy image samples and missing electron microscopy image samples are one-to-one correspondences and appear in pairs.

[0097] Step 704: In the first iteration, the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample is input into the denoising network for denoising to obtain the first reconstructed feature map; the first reconstructed feature map and the masked electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain the first reconstructed image.

[0098] The computer device multiplies the missing electron microscope image sample and the image mask sample to obtain the masked electron microscope image, which is then input into a recurrent neural network (RNN) for k iterations to reconstruct the image. The RNN includes a denoising network and a data consistency network, where k is a positive integer. The denoising network denoises the input image, and the data consistency network optimizes the input image using conjugate gradients.

[0099] For example, the optimization processing algorithm for the data consistency network may be at least one of the following: Adaptive Gradient (AdaGrad), Stochastic Gradient Descent (SGD), Standard Momentum Optimization Algorithm, and Root Mean Square Propagation (RMSprop). This application does not limit the specific algorithm used.

[0100] Exemplarily, the network structure of the denoiser network is based on at least one of Convolutional Neural Network (CNN), ResNet, VGG network (proposed by the Visual Geometry Group), and the lightweight deep neural network MobileNet. This application embodiment does not limit the specific network structure. In this embodiment, the network structure of the denoiser network is CNN.

[0101] In one possible implementation, during the first iteration, the computer device inputs the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into a denoising network for noise reduction, resulting in a first reconstructed feature map. With the first reconstructed feature map obtained, the computer device inputs both the first reconstructed feature map and the masked electron microscope image into a data consistency network for conjugate gradient optimization, resulting in a first reconstructed image. The reconstructed feature map represents the result obtained after noise reduction by the denoising network. The reconstructed image represents the result obtained after the reconstructed feature map undergoes conjugate gradient optimization by the data consistency network.

[0102] Step 706: During the i-th iteration, the (i-1)-th reconstructed image is input into the denoising network for denoising to obtain the i-th reconstructed feature map; the i-th reconstructed feature map and the mask electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain the i-th reconstructed image.

[0103] In one possible implementation, during the i-th iteration, the computer device inputs the (i-1)-th reconstructed image into a denoising network for denoising to obtain the i-th reconstructed feature map; after obtaining the i-th reconstructed feature map, the computer device inputs the i-th reconstructed feature map and the mask electron microscope image into a data consistency network for conjugate gradient optimization to obtain the i-th reconstructed image.

[0104] The noise reduction network consists of n three-dimensional convolutional layers, where n is an integer greater than 1.

[0105] In one possible implementation, during the i-th iteration, the computer device inputs the (i-1)-th reconstructed image into the denoising network. During the first convolution in the denoising network, the computer device inputs the (i-1)-th reconstructed image into the first of n three-dimensional convolutional layers for convolution, obtaining the first convolution result. The convolution result represents the result obtained after convolution processing by the three-dimensional convolutional layers in the denoising network.

[0106] During the j-th convolution process, the (j-1)-th convolution result is input into the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution to obtain the j-th convolution result, where j is an integer greater than 1 and not greater than n.

[0107] When j equals n, the result of the j-th convolution is output as the i-th reconstructed feature map.

[0108] In one possible implementation, among the n three-dimensional convolutional layers, the 1st to n-1th three-dimensional convolutional layers include a three-dimensional convolutional kernel layer, a batch normalization layer, and a rectified linear unit (ReLU) activation layer; the nth three-dimensional convolutional layer includes a three-dimensional convolutional kernel layer and a batch normalization layer.

[0109] In one possible implementation, during the j-th convolution, the computer device inputs the (j-1)-th convolution result into the j-th three-dimensional convolutional layer among n three-dimensional convolutional layers for convolution. If j is greater than 1 and less than n, during the j-th convolution, the (j-1)-th convolution result is input into the three-dimensional convolutional kernel layer among the j-th three-dimensional convolutional layers for convolution, thus obtaining the j-th convolutional feature. The convolutional feature is used to represent the result obtained by processing the convolution result through the three-dimensional convolutional kernel layer in the three-dimensional convolutional layer.

[0110] The computer device inputs the j-th convolutional feature into the batch normalization layer of the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for normalization processing, and obtains the normalized j-th convolutional feature.

[0111] The computer device inputs the normalized j-th convolutional feature into the ReLU activation layer of the j-th three-dimensional convolutional layer in the n three-dimensional convolutional layers for activation processing, and obtains the activated j-th convolutional feature.

[0112] In one possible implementation, during the j-th convolution, the computer device inputs the (j-1)-th convolution result into the j-th three-dimensional convolutional layer among n three-dimensional convolutional layers for convolution. When j equals n, during the j-th convolution, the (j-1)-th convolution result is input into the three-dimensional convolutional kernel layer of the n-th three-dimensional convolutional layer among n three-dimensional convolutional layers for convolution, thus obtaining the n-th convolutional feature.

[0113] The nth convolutional feature is input into the batch normalization layer of the nth three-dimensional convolutional layer for normalization processing, resulting in the normalized nth convolutional feature. The normalized nth convolutional feature is then output as the i-th reconstructed feature map.

[0114] Step 708: When i equals k, output the i-th reconstructed image as a reconstructed electron microscope image.

[0115] In one possible implementation, when i equals k, i.e., in the last iteration of the current training process, the computer device outputs the i-th reconstructed image as a reconstructed electron microscope image.

[0116] The computer device inputs the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into the denoising network and the data consistency network for k iterations of reconstruction, and outputs the reconstructed electron microscope image corresponding to the missing electron microscope image sample.

[0117] Step 710: Train the model parameters of the reconstruction model based on the error between the complete electron microscope image sample and the reconstructed electron microscope image.

[0118] The computer equipment trains the model parameters of the reconstruction model based on the error between the complete electron microscope image sample and the reconstructed electron microscope image. The reconstructed model after training is used to process the input missing electron microscope image and image mask.

[0119] For example, the error function for calculating the difference between the complete electron microscope image sample and the reconstructed electron microscope image can be at least one of the following: Euclidean distance function, Manhattan distance function, Chebyshev distance function, Minkowski distance function, standardized Euclidean distance function, Mahalanobis distance function, included angle cosine function, Hamming distance function, and information entropy function. This application embodiment does not limit this.

[0120] In one possible implementation, the computer device calculates the mean squared error of the reconstruction model based on complete electron microscope image samples and reconstructed electron microscope images, updates the model parameters of the reconstruction model according to the mean squared error, and trains the model parameters of the reconstruction model. The reconstructed model after training is used to process the input missing electron microscope images and image masks.

[0121] The mean square error of the reconstructed model can be obtained by the following formula:

[0122]

[0123] In the formula: x rec Represented as a reconstructed electron microscope image; x full denoted as a complete electron microscopy image sample; N represents the number of samples in the training set that are missing electron microscopy image samples.

[0124] In summary, the solution shown in this application involves acquiring missing electron microscope image samples, image mask samples, and complete electron microscope image samples in a three-dimensional image format. The computer device multiplies the missing electron microscope image samples and the image mask samples to obtain a masked electron microscope image, which is then input into n three-dimensional convolutional layers in the noise reduction network for convolution processing to obtain a reconstructed feature map.

[0125] The computer equipment inputs the reconstructed feature map and mask electron microscope image output by the noise reduction network into the data consistency network for conjugate gradient optimization processing to obtain the reconstructed image.

[0126] The computer device multiplies the acquired missing electron microscope image sample with the image mask sample to obtain the masked electron microscope image. After k iterations in the denoising network and data consistency network of the recurrent neural network, the reconstructed electron microscope image is obtained.

[0127] The computer equipment trains the model parameters of the reconstruction model based on the error between the complete electron microscope image sample and the reconstructed electron microscope image, thus obtaining the trained reconstruction model. The three-dimensional electron microscope image reconstruction model training method provided in this application, which obtains a three-dimensional electron microscope image reconstruction model through k iterations, can better recover the information lost in cryo-electron tomography imaging data with low signal-to-noise ratio.

[0128] Figure 8 A flowchart illustrating a method for reconstructing three-dimensional electron microscope images according to an exemplary embodiment of this application is shown. This method can be executed by a computing device, which can be implemented as follows: Figure 4 The reconstruction device 420 for the three-dimensional electron microscope image shown, such as Figure 8 As shown, the method for reconstructing this three-dimensional electron microscope image may include the following steps:

[0129] Step 802: Obtain the missing electron microscope image and image mask in a three-dimensional image format.

[0130] Computer equipment acquires missing electron microscopy images and image masks in 3D image format. Missing electron microscopy images are cryo-electron microscopy images with missing data.

[0131] An image mask is used to mask invalid pixels in a missing electron microscope image sample. For example, an image mask is used to describe data loss in an electron microscope image; or, an image mask is used to describe data loss caused by the limited scanning angle range during cryo-electron tomography of a sample.

[0132] Step 804: In the first iteration, the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask is input into the denoising network for denoising to obtain the first reconstructed feature map; the first reconstructed feature map and the masked electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain the first reconstructed image.

[0133] The computer device multiplies the missing electron microscope image with an image mask to obtain a masked electron microscope image, which is then input into a recurrent neural network (RNN) for k iterations to reconstruct the image. The RNN includes a denoising network and a data consistency network, where k is a positive integer. The denoising network denoises the input image, and the data consistency network optimizes the input image using conjugate gradients.

[0134] In one possible implementation, during the first iteration, the computer device inputs the masked electron microscope image (obtained by multiplying the missing electron microscope image and the image mask) into a denoising network for noise reduction, resulting in a first reconstructed feature map. With the first reconstructed feature map obtained, the computer device inputs both the first reconstructed feature map and the masked electron microscope image into a data consistency network for conjugate gradient optimization, resulting in a first reconstructed image. The reconstructed feature map represents the result obtained after noise reduction by the denoising network. The reconstructed image represents the result obtained after conjugate gradient optimization of the reconstructed feature map by the data consistency network.

[0135] Step 806: During the i-th iteration, the (i-1)-th reconstructed image is input into the denoising network for denoising to obtain the i-th reconstructed feature map; the i-th reconstructed feature map and the mask electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain the i-th reconstructed image.

[0136] In one possible implementation, during the i-th iteration, the computer device inputs the (i-1)-th reconstructed image into a denoising network for denoising to obtain the i-th reconstructed feature map; after obtaining the i-th reconstructed feature map, the computer device inputs the i-th reconstructed feature map and the mask electron microscope image into a data consistency network for conjugate gradient optimization to obtain the i-th reconstructed image.

[0137] The noise reduction network consists of n three-dimensional convolutional layers, where n is an integer greater than 1.

[0138] In one possible implementation, during the i-th iteration, the computer device inputs the (i-1)-th reconstructed image into the denoising network. During the first convolution in the denoising network, the computer device inputs the (i-1)-th reconstructed image into the first of n three-dimensional convolutional layers for convolution, obtaining the first convolution result. The convolution result represents the result obtained by processing the input image through the three-dimensional convolutional layers in the denoising network.

[0139] During the j-th convolution process, the (j-1)-th convolution result is input into the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution to obtain the j-th convolution result, where j is an integer greater than 1 and not greater than n.

[0140] When j equals n, the result of the j-th convolution is output as the i-th reconstructed feature map.

[0141] In one possible implementation, among the n three-dimensional convolutional layers, the 1st to n-1th three-dimensional convolutional layers include a three-dimensional convolutional kernel layer, a batch normalization layer, and a ReLU activation layer; the nth three-dimensional convolutional layer includes a three-dimensional convolutional kernel layer and a batch normalization layer.

[0142] In one possible implementation, during the j-th convolution, the computer device inputs the (j-1)-th convolution result into the j-th three-dimensional convolutional layer among n three-dimensional convolutional layers for convolution. If j is greater than 1 and less than n, during the j-th convolution, the (j-1)-th convolution result is input into the three-dimensional convolutional kernel layer among the j-th three-dimensional convolutional layers for convolution, thus obtaining the j-th convolutional feature. The convolutional feature is used to represent the result obtained by processing the convolution result through the three-dimensional convolutional kernel layer in the three-dimensional convolutional layer.

[0143] The computer device inputs the j-th convolutional feature into the batch normalization layer in the j-th three-dimensional convolutional layer of the n three-dimensional convolutional layers for normalization processing to obtain the normalized j-th convolutional feature.

[0144] The computer device inputs the normalized j-th convolutional feature into the ReLU activation layer of the j-th three-dimensional convolutional layer among n three-dimensional convolutional layers for activation processing, and obtains the activated j-th convolutional feature.

[0145] In one possible implementation, during the j-th convolution, the computer device inputs the (j-1)-th convolution result into the j-th three-dimensional convolutional layer among n three-dimensional convolutional layers for convolution. When j equals n, during the j-th convolution, the (j-1)-th convolution result is input into the three-dimensional convolutional kernel layer of the n-th three-dimensional convolutional layer among n three-dimensional convolutional layers for convolution, thus obtaining the n-th convolutional feature.

[0146] The nth convolutional feature is input into the batch normalization layer of the nth three-dimensional convolutional layer for normalization processing, resulting in the normalized nth convolutional feature. The normalized nth convolutional feature is then output as the i-th reconstructed feature map.

[0147] Step 808: When i equals k, output the i-th reconstructed image as a reconstructed electron microscope image.

[0148] In one possible implementation, when i equals k, the computer device outputs the i-th reconstructed image as a reconstructed electron microscope image.

[0149] The computer device inputs the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask into the denoising network and the data consistency network for k iterations of reconstruction, and outputs the reconstructed electron microscope image corresponding to the missing electron microscope image.

[0150] Step 810: Output the reconstructed electron microscope image.

[0151] In summary, the solution presented in this application involves acquiring a missing electron microscope image and an image mask sample in a three-dimensional image format; inputting the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask into a trained reconstruction model for k iterations to obtain a reconstructed electron microscope image; and finally, outputting the reconstructed electron microscope image. The three-dimensional electron microscope image reconstruction method provided in this application can effectively recover information lost in cryo-electron tomography data with low signal-to-noise ratios.

[0152] Figure 9 A block diagram of a three-dimensional electron microscope image reconstruction model training apparatus is shown in an exemplary embodiment of this application. This apparatus can be used to implement, for example... Figure 2 or Figure 7 All or part of the steps in the method shown. For example... Figure 9 As shown, the training device for the reconstruction model of the three-dimensional electron microscope images includes:

[0153] The sample acquisition module 901 is used to acquire missing electron microscope image samples, image mask samples, and complete electron microscope image samples in a three-dimensional image format. The missing electron microscope image samples are cryo-electron microscope images with missing data. The image mask samples are used to mask invalid pixels in the missing electron microscope image samples. The complete electron microscope image samples are cryo-electron microscope images without missing data.

[0154] The sample iteration module 902 is used to input the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into the reconstruction model for k loop iterations to reconstruct the reconstructed electron microscope image.

[0155] Training module 903 is used to train the model parameters of the reconstruction model based on the error between the complete electron microscope image samples and the reconstructed electron microscope images.

[0156] In one possible implementation, the reconstruction model is a recurrent neural network;

[0157] The sample iteration module 902 is also used to input the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into the recurrent neural network for k iterations of reconstruction to obtain the reconstructed electron microscope image, where k is an integer greater than 0.

[0158] In one possible implementation, the recurrent neural network includes a denoising network and a data consistency network;

[0159] The sample iteration module 902 is also used in the first iteration process to input the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into the denoising network for denoising to obtain the first reconstructed feature map; and to input the first reconstructed feature map and the masked electron microscope image into the data consistency network for conjugate gradient optimization to obtain the first reconstructed image.

[0160] The sample iteration module 902 is also used in the i-th iteration process to input the mask electron microscope image obtained by multiplying the i-1 reconstructed image and the image mask sample into the denoising network for denoising to obtain the i-th reconstructed feature map; and to input the i-th reconstructed feature map and the mask electron microscope image into the data consistency network for conjugate gradient optimization to obtain the i-th reconstructed image; i is an integer greater than 1 and not greater than k.

[0161] When i equals k, the i-th reconstructed image is output as a reconstructed electron microscope image.

[0162] In one possible implementation, the noise reduction network includes: n three-dimensional convolutional layers;

[0163] The sample iteration module 902 is also used in the first convolution process to input the mask electron microscope image obtained by multiplying the (i-1)th reconstructed image and the image mask sample into the first three-dimensional convolution layer among the n three-dimensional convolution layers for convolution to obtain the first convolution result.

[0164] The sample iteration module 902 is also used to input the (j-1)th convolution result into the jth three-dimensional convolution layer among the n three-dimensional convolution layers during the jth convolution process to obtain the jth convolution result, where j is an integer greater than 1 and not greater than n; n is an integer greater than 1.

[0165] When j equals n, the j-th convolution result is output as the i-th reconstructed feature map.

[0166] In one possible implementation, 1 to n-1 of the n three-dimensional convolutional layers include a three-dimensional convolutional kernel layer, a batch normalization layer, and a modified linear unit (ReLU) activation layer.

[0167] The sample iteration module 902 is also used to input the (j-1)th convolution result into the three-dimensional convolution kernel layer of the jth three-dimensional convolution layer in the jth three-dimensional convolution layer during the jth convolution process when j is greater than 1 and less than n, so as to obtain the jth convolution feature.

[0168] The sample iteration module 902 is also used to input the j-th convolutional feature into the batch normalization layer in the n three-dimensional convolutional layers for normalization processing to obtain the normalized j-th convolutional feature;

[0169] The sample iteration module 902 is also used to input the normalized j-th convolutional feature into the ReLU activation layer in the n three-dimensional convolutional layers for activation processing, so as to obtain the activated j-th convolutional feature.

[0170] In one possible implementation, the nth three-dimensional convolutional layer in the n three-dimensional convolutional layers includes a three-dimensional convolutional kernel layer and a batch normalization layer;

[0171] The sample iteration module 902 is also used to input the (j-1)th convolution result into the nth three-dimensional convolution kernel layer in the n three-dimensional convolution layers during the j-th convolution process when j equals n, so as to obtain the nth convolution feature;

[0172] The sample iteration module 902 is also used to input the nth convolutional feature into the nth batch normalization layer in the n three-dimensional convolutional layers for normalization processing, to obtain the normalized nth convolutional feature, and to output the normalized nth convolutional feature as the i-th reconstructed feature map.

[0173] In one possible implementation, the training module 903 is used to calculate the mean square error of the reconstruction model based on the complete electron microscope image samples and the reconstructed electron microscope image, update the model parameters of the reconstruction model according to the mean square error, and train the model parameters of the reconstruction model.

[0174] Figure 10 The diagram illustrates a block diagram of a three-dimensional electron microscope image reconstruction apparatus according to an exemplary embodiment of this application. This apparatus can be used to achieve, for example... Figure 3 or Figure 8 All or part of the steps in the method shown. For example... Figure 10 As shown, the reconstruction apparatus for the three-dimensional electron microscope image includes:

[0175] The acquisition module 1001 is used to acquire missing electron microscopy images and image masks in a three-dimensional image format. The missing electron microscopy images are cryo-electron microscopy images with missing data.

[0176] The iteration module 1002 is used to input the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask into the reconstruction model to perform k loop iterations to reconstruct the reconstructed electron microscope image.

[0177] Output module 1003 is used to output reconstructed electron microscope images.

[0178] In one possible implementation, the reconstruction model is a recurrent neural network;

[0179] The iteration module 1002 is also used to input the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask into the recurrent neural network for k iterations of reconstruction to obtain the reconstructed electron microscope image, where k is an integer greater than 0.

[0180] In one possible implementation, the recurrent neural network includes a denoising network and a data consistency network;

[0181] The iteration module 1002 is also used in the first iteration process to input the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask into the denoising network for denoising to obtain the first reconstructed feature map; and to input the first reconstructed feature map and the masked electron microscope image into the data consistency network for conjugate gradient optimization to obtain the first reconstructed image.

[0182] The iteration module 1002 is also used in the i-th iteration process to input the (i-1)-th reconstructed image into the denoising network for denoising to obtain the i-th reconstructed feature map; and to input the i-th reconstructed feature map and the mask electron microscope image into the data consistency network for conjugate gradient optimization to obtain the i-th reconstructed image; i is an integer greater than 1 and not greater than k;

[0183] When i equals k, the i-th reconstructed image is output as the reconstructed electron microscope image.

[0184] In one possible implementation, the noise reduction network includes: n three-dimensional convolutional layers;

[0185] The iteration module 1002 is also used to input the (i-1)th reconstructed image into the first three-dimensional convolutional layer among the n three-dimensional convolutional layers during the first convolution process to obtain the first convolution result;

[0186] The iteration module 1002 is also used to input the (j-1)th convolution result into the jth three-dimensional convolution layer among the n three-dimensional convolution layers during the jth convolution process to obtain the jth convolution result, where j is an integer greater than 1 and not greater than n; n is an integer greater than 1.

[0187] When j equals n, the j-th convolution result is output as the i-th reconstructed feature map.

[0188] In one possible implementation, 1 to n-1 of the n three-dimensional convolutional layers include a three-dimensional convolutional kernel layer, a batch normalization layer, and a ReLU activation layer;

[0189] The iteration module 1002 is further configured to, when j is greater than 1 and less than n, during the j-th convolution process, input the (j-1)-th convolution result to the three-dimensional convolution kernel layer of the j-th three-dimensional convolution layer in the n three-dimensional convolution layers for convolution to obtain the j-th convolution feature; input the j-th convolution feature to the batch normalization layer of the n three-dimensional convolution layers for normalization processing to obtain the normalized j-th convolution feature; and input the normalized j-th convolution feature to the ReLU activation layer of the n three-dimensional convolution layers for activation processing to obtain the activated j-th convolution feature.

[0190] In one possible implementation, the nth three-dimensional convolutional layer in the n three-dimensional convolutional layers includes a three-dimensional convolutional kernel layer and a batch normalization layer;

[0191] The iteration module 1002 is also used to input the (j-1)th convolution result into the nth three-dimensional convolution kernel layer in the n three-dimensional convolution layers during the j-th convolution process when j equals n, so as to obtain the nth convolution feature;

[0192] The iteration module 1002 is further configured to input the nth convolutional feature into the nth batch normalization layer of the n three-dimensional convolutional layers for normalization processing to obtain the normalized nth convolutional feature, and output the normalized nth convolutional feature as the i-th reconstructed feature map.

[0193] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the three-dimensional electron microscope image reconstruction model training method provided in the above-described method embodiments.

[0194] Alternatively, the computer device is a server. For example, Figure 11 This is a structural block diagram of a server provided in an exemplary embodiment of this application.

[0195] Typically, server 1100 includes a processor 1101 and memory 1102.

[0196] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0197] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one instruction, which is executed by the processor 1101 to implement the video data tracking method provided in the method embodiments of this application.

[0198] In some embodiments, server 1100 may optionally include an input interface 1103 and an output interface 1104. Processor 1101, memory 1102, and input interface 1103 and output interface 1104 can be connected via a bus or signal lines. Various peripheral devices can be connected to input interface 1103 and output interface 1104 via a bus, signal lines, or circuit board. Input interface 1103 and output interface 1104 can be used to connect at least one input / output (I / O) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102, and input interface 1103 and output interface 1104 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102, and input interface 1103 and output interface 1104 can be implemented on separate chips or circuit boards, and this application embodiment does not limit this.

[0199] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on server 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0200] In one exemplary embodiment, a computer-readable storage medium is also provided for storing at least one computer instruction, which is loaded and executed by a processor to implement all or part of the steps in the above-described method for training a reconstruction model of a three-dimensional electron microscope image, or all or part of the steps in the method for reconstructing a three-dimensional electron microscope image. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0201] In one exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform all or part of the steps in the above-described method for training a reconstruction model of a three-dimensional electron microscope image, or all or part of the steps in the method for reconstructing a three-dimensional electron microscope image.

[0202] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0203] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for training a reconstruction model of a three-dimensional electron microscope image, characterized in that, The method includes: The process involves acquiring missing electron microscope (TEM) image samples, image mask samples, and complete TEM image samples in a 3D image format. The missing TEM image samples are cryo-electron microscopy (cryo-EM) images with missing data. The image mask samples are used to mask invalid pixels in the missing TEM image samples. The complete TEM image samples are cryo-EM images without missing data. The image mask samples are used to describe the data loss caused by the limited scanning angle range when using cryo-electron tomography to scan samples. The masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample is input into the reconstruction model for k iterations to obtain the reconstructed electron microscope image, where k is an integer greater than 0. The reconstruction model is a recurrent neural network, which includes a denoising network and a data consistency network. In the first iteration, the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample is input into the denoising network for denoising to obtain the first reconstructed feature map. The first reconstructed feature map and the masked electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain the first reconstructed image. In the i-th iteration, the (i-1)-th reconstructed image is input into the denoising network for denoising to obtain the i-th reconstructed feature map. The i-th reconstructed feature map and the masked electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain the i-th reconstructed image, where i is an integer greater than 1 and not greater than k. When i equals k, the i-th reconstructed image is output as the reconstructed electron microscope image. The model parameters of the reconstruction model are trained based on the error between the complete electron microscope image sample and the reconstructed electron microscope image.

2. The method according to claim 1, characterized in that, The noise reduction network includes: n three-dimensional convolutional layers; The step of inputting the (i-1)th reconstructed image into the denoising network for denoising to obtain the i-th reconstructed feature map includes: In the first convolution process, the (i-1)th reconstructed image is input into the first three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution to obtain the first convolution result; During the j-th convolution process, the (j-1)-th convolution result is input into the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution to obtain the j-th convolution result, where j is an integer greater than 1 and not greater than n; and n is an integer greater than 1. When j equals n, the j-th convolution result is output as the i-th reconstructed feature map.

3. The method according to claim 2, characterized in that, The n three-dimensional convolutional layers, from 1 to n-1, include three-dimensional convolutional kernel layers, batch normalization layers, and modified linear unit (ReLU) activation layers. In the j-th convolution process, the (j-1)-th convolution result is input into the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution to obtain the j-th convolution result, including: When j is greater than 1 and less than n, during the j-th convolution process, the (j-1)-th convolution result is input into the three-dimensional convolution kernel layer of the j-th three-dimensional convolution layer among the n three-dimensional convolution layers to perform convolution and obtain the j-th convolution feature; The j-th convolutional feature is input into the batch normalization layer of the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for normalization processing to obtain the normalized j-th convolutional feature. The normalized j-th convolutional feature is input into the ReLU activation layer of the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for activation processing, to obtain the activated j-th convolutional feature.

4. The method according to claim 2, characterized in that, The nth three-dimensional convolutional layer in the n three-dimensional convolutional layers includes a three-dimensional convolutional kernel layer and a batch normalization layer; When j equals n, outputting the j-th convolution result as the i-th reconstructed feature map includes: When j equals n, during the j-th convolution process, the (j-1)-th convolution result is input into the nth three-dimensional convolution kernel layer among the n three-dimensional convolution layers for convolution to obtain the nth convolution feature; The nth convolutional feature is input into the nth batch normalization layer of the n three-dimensional convolutional layers for normalization processing to obtain the normalized nth convolutional feature. The normalized nth convolutional feature is then output as the i-th reconstructed feature map.

5. The method according to any one of claims 1 to 4, characterized in that, The step of training the model parameters of the reconstruction model based on the error between the complete electron microscope image sample and the reconstructed electron microscope image includes: Based on the complete electron microscope image sample and the reconstructed electron microscope image, the mean square error of the reconstruction model is calculated, the model parameters of the reconstruction model are updated according to the mean square error, and the model parameters of the reconstruction model are trained.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Acquire missing electron microscopy images and image masks in a three-dimensional image format, wherein the missing electron microscopy images are cryo-electron microscopy images with missing data; The masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask is input into the trained reconstruction model for k iterations of reconstruction to obtain the reconstructed electron microscope image. Output the reconstructed electron microscope image.

7. A method for reconstructing a three-dimensional electron microscope image, characterized in that, Applied in a computer device, the computer device running a reconstruction model trained by the method as described in any one of claims 1 to 6, the method comprising: Acquire missing electron microscopy images and image masks in a three-dimensional image format. The missing electron microscopy images are cryo-electron microscopy images with missing data. The image mask is used to describe the data loss caused by the limited scanning angle range when cryo-electron tomography scans the sample. The masked electron microscope image, obtained by multiplying the missing electron microscope image and the image mask, is input into the reconstruction model for k iterations to obtain a reconstructed electron microscope image, where k is an integer greater than 0. The reconstruction model is a recurrent neural network, which includes a denoising network and a data consistency network. In the first iteration, the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask is input into the denoising network for denoising to obtain a first reconstructed feature map. The first reconstructed feature map and the masked electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain a first reconstructed image. In the i-th iteration, the (i-1)-th reconstructed image is input into the denoising network for denoising to obtain an i-th reconstructed feature map. The i-th reconstructed feature map and the masked electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain an i-th reconstructed image, where i is an integer greater than 1 and not greater than k. When i equals k, the i-th reconstructed image is output as the reconstructed electron microscope image. Output the reconstructed electron microscope image.

8. The method according to claim 7, characterized in that, The noise reduction network includes: n three-dimensional convolutional layers; The step of inputting the (i-1)th reconstructed image into the denoising network for denoising to obtain the i-th reconstructed feature map includes: In the first convolution process, the (i-1)th reconstructed image is input into the first three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution to obtain the first convolution result; During the j-th convolution process, the (j-1)-th convolution result is input into the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution to obtain the j-th convolution result, where j is an integer greater than 1 and not greater than n; and n is an integer greater than 1. When j equals n, the j-th convolution result is output as the i-th reconstructed feature map.

9. The method according to claim 8, characterized in that, The n three-dimensional convolutional layers, from 1 to n-1, include three-dimensional convolutional kernel layers, batch normalization layers, and modified linear unit (ReLU) activation layers. In the j-th convolution process, the (j-1)-th convolution result is input into the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for convolution to obtain the j-th convolution result, including: When j is greater than 1 and less than n, during the j-th convolution process, the (j-1)-th convolution result is input into the three-dimensional convolution kernel layer of the j-th three-dimensional convolution layer among the n three-dimensional convolution layers to perform convolution and obtain the j-th convolution feature; The j-th convolutional feature is input into the batch normalization layer of the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for normalization processing to obtain the normalized j-th convolutional feature. The normalized j-th convolutional feature is input into the ReLU activation layer of the j-th three-dimensional convolutional layer among the n three-dimensional convolutional layers for activation processing, to obtain the activated j-th convolutional feature.

10. The method according to claim 8, characterized in that, The nth three-dimensional convolutional layer in the n three-dimensional convolutional layers includes a three-dimensional convolutional kernel layer and a batch normalization layer; When j equals n, outputting the j-th convolution result as the i-th reconstructed feature map includes: When j equals n, during the j-th convolution process, the (j-1)-th convolution result is input into the nth three-dimensional convolution kernel layer among the n three-dimensional convolution layers for convolution to obtain the nth convolution feature; The nth convolutional feature is input into the nth batch normalization layer of the n three-dimensional convolutional layers for normalization processing to obtain the normalized nth convolutional feature. The normalized nth convolutional feature is then output as the i-th reconstructed feature map.

11. A training device for a reconstruction model of a three-dimensional electron microscope image, characterized in that, The device includes: The sample acquisition module is used to acquire missing electron microscope image samples, image mask samples, and complete electron microscope image samples in a three-dimensional image format. The missing electron microscope image samples are cryo-electron microscope images with missing data. The image mask samples are used to mask invalid pixels in the missing electron microscope image samples. The complete electron microscope image samples are cryo-electron microscope images without missing data. The image mask samples are used to describe the data loss caused by the limited scanning angle range when cryo-electron tomography scans the sample. The sample iteration module is used to input the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample into the reconstruction model for k iterations to obtain the reconstructed electron microscope image, where k is an integer greater than 0; the reconstruction model is a recurrent neural network, which includes a denoising network and a data consistency network; in the first iteration, the masked electron microscope image obtained by multiplying the missing electron microscope image sample and the image mask sample is input into the denoising network for noise reduction to obtain the first reconstructed feature. Figure 1 shows the process of inputting the first reconstructed feature map and the masked electron microscope image into the data consistency network for conjugate gradient optimization to obtain the first reconstructed image; during the i-th iteration, the (i-1)-th reconstructed image is input into the denoising network for noise reduction to obtain the i-th reconstructed feature map; the i-th reconstructed feature map and the masked electron microscope image are then input into the data consistency network for conjugate gradient optimization to obtain the i-th reconstructed image; i is an integer greater than 1 and not greater than k; when i equals k, the i-th reconstructed image is output as the reconstructed electron microscope image. The training module is used to train the model parameters of the reconstruction model based on the error between the complete electron microscope image sample and the reconstructed electron microscope image.

12. A device for reconstructing three-dimensional electron microscope images, characterized in that, The device operates a reconstruction model trained by the training device according to claim 11, the device comprising: The acquisition module is used to acquire missing electron microscopy images and image masks in a three-dimensional image format. The missing electron microscopy images are cryo-electron microscopy images with missing data. The image mask is used to describe the data loss caused by the limited scanning angle range when cryo-electron tomography scans the sample. An iterative module is used to input the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask into the reconstruction model for k iterations of reconstruction to obtain a reconstructed electron microscope image, where k is an integer greater than 0. The reconstruction model is a recurrent neural network, which includes a denoising network and a data consistency network. In the first iteration, the masked electron microscope image obtained by multiplying the missing electron microscope image and the image mask is input into the denoising network for denoising to obtain a first reconstructed feature map. The first reconstructed feature map and the masked electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain a first reconstructed image. In the i-th iteration, the (i-1)-th reconstructed image is input into the denoising network for denoising to obtain an i-th reconstructed feature map. The i-th reconstructed feature map and the masked electron microscope image are input into the data consistency network for conjugate gradient optimization to obtain an i-th reconstructed image, where i is an integer greater than 1 and not greater than k. When i equals k, the i-th reconstructed image is output as the reconstructed electron microscope image. The output module is used to output the reconstructed electron microscope image.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the method for training a reconstruction model of a three-dimensional electron microscope image as described in any one of claims 1 to 6, or the method for reconstructing a three-dimensional electron microscope image as described in any one of claims 7 to 10.

14. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the method for training a reconstruction model of a three-dimensional electron microscope image as described in any one of claims 1 to 6, or the method for reconstructing a three-dimensional electron microscope image as described in any one of claims 7 to 10.

15. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the computer instructions are read from and executed by a processor of a computer device, causing the computer device to perform the method for training a reconstruction model of a three-dimensional electron microscope image as described in any one of claims 1 to 6, or the method for reconstructing a three-dimensional electron microscope image as described in any one of claims 7 to 10.

Images