A method for reconstructing volume data of electron microscope

Abstract

The present invention provides an electron microscope volume data reconstruction method. The network architecture of the method includes a generator X and a generator Y, a discriminator X and a discriminator Y, a segmenter S and a discriminator E, so as to realize converting volume data X into volume data fake-Y; converting volume data Y into first volume data fake-X; then inputting fake-X and the discriminator E to distinguish the true from the false, and then inputting the segmenter S to perform data segmentation processing to obtain first segmented image data, and then judging the source of the segmented data by the discriminator E, and finally determining that the segmented data obtained by fake-X is reconstructed into microstructure data.

Description

Technical Field

[0001] The present invention relates to the fields of digital image processing and deep learning, and in particular to a method for reconstructing volume data of an electron microscope. Background Art

[0002] Large-scale electron microscope volume data is the data support for reconstructing the microstructure of some objects, such as protein structure reconstruction, neuron reconstruction, cell structure reconstruction, etc. Volume data is obtained by cutting the sample into extremely thin samples and stacking them through electron microscope scanning. However, since the slices are very fragile, it is almost impossible to obtain more than a large number of continuous and flawless slices and perform imaging. Each slice stack will contain a variety of image defects, such as inconsistent contrast and brightness, folding, tearing, image shrinkage during imaging, lens distortion, slice loss, slice contamination, and dust particles. This will leave blanks in the complete image and affect the reconstruction of the microstructure. And because it is difficult to retrieve lost slices and it is difficult to restore the contaminated slices, it is difficult to solve the problem in data acquisition.

[0003] In order to correctly reconstruct these microscopic structures, these ultra-high resolution images need to be segmented effectively and accurately. However, due to the complexity of the microscopic structures, the number of these structures and the large number of images in ultra-high resolution electron microscope images, and the volume data usually contains a large number of stacked images, manual labeling is inefficient and prone to errors.

[0004] Manual labeling requires accurate segmentation of each layer of the image. Since the contours of the microstructure are usually irregular, the segmentation labels usually need to be accurate to a single pixel, so manual segmentation cannot be performed quickly. In addition, each layer of the volume data contains a lot of similar information, so manual labeling is time-consuming and labor-intensive.

[0005] Because slices are very fragile, it is almost impossible to obtain a large number of continuous and flawless slices for imaging. Each slice stack will contain multiple image defects. In addition, it is difficult to resolve the data defect problem in data acquisition because it is difficult to retrieve lost slices and restore damaged slices.

[0006] The reconstruction process of the microscopic structure of matter usually requires manual segmentation, but in ultra-high-resolution electron microscope images, these structures are numerous and the number of images is large, and the volume data usually contains a large number of stacked images. Manual labeling is inefficient and prone to errors.

[0007] On the one hand, in the process of studying large-scale electron microscope volume data used to reconstruct microstructures, it was found that there are usually many defects in the data, such as imaging blur, folding, tearing, image shrinkage during imaging, lens distortion, slice loss, slice contamination, and dust particles. These problems will affect the quality of the subsequent algorithm for data segmentation and affect the results of structural reconstruction.

[0008] On the other hand, when investigating image segmentation algorithms, it was found that the image segmentation method based on affinity graph learning outperformed other image segmentation methods in the electron microscope image segmentation task with certain advantages. It has high segmentation quality and is suitable for dense segmentation tasks with a large number of similar structures, such as neuron segmentation. Therefore, the image segmentation method based on affinity graph learning can accurately segment the data and reduce the manual workload. Summary of the invention

[0009] In order to solve the above technical problems, based on the above research mentioned in the present invention, the present invention provides a method for reconstructing volume data of an electron microscope, characterized in that the method comprises:

[0010] Preprocess the original volume Y using a pre-trained voxel enhancement algorithm to obtain the first volume data fake-X;

[0011] The volume data X and the first volume data fake-X are used as inputs of a pre-trained first segmenter for segmentation processing, and a discriminator E is used to perform discriminant processing on the segmented data, and the data obtained from fake-X is reconstructed into microstructure data.

[0012] Preferably, the pre-trained voxel enhancement algorithm is an improved recurrent generative adversarial network algorithm.

[0013] Further preferably, the network architecture of the improved cyclic generative adversarial network algorithm includes:

[0014] A generator X, configured to convert the volume data X into volume data fake-Y;

[0015] A generator Y, configured to convert the volume data Y into the first volume data fake-X;

[0016] Discriminator X and discriminator Y are used to determine whether the input data is true or false;

[0017] A segmenter S, used for segmenting the input data to obtain first segmented image data;

[0018] The discriminator E is used to discriminate the source of the first segmented image data.

[0019] Further preferably, the backbone network of the generator X and the generator Y adopts a U-Net neural network.

[0020] Further preferably, the backbone network of the discriminator X, the discriminator Y and the discriminator E adopts a multi-layer convolutional neural network.

[0021] Further preferably, the method further comprises:

[0022] The loss function when training the generator Y is:

[0023] |G x (G Y (Y))-Y|+(D X (G Y (Y))-1) 2 +(D S (S(G Y (Y)))-1) 2

[0024] The loss function when training the discriminator E is:

[0025] (D S (S(G Y (Y)))-0) 2 +(D s (S(X))-1) 2

[0026] Wherein, X represents the volume data X, Y represents the original volume Y, G represents the generator, D represents the discriminator, and S represents the segmenter.

[0027] Preferably, the method further includes: the first segmenter is an image segmentation algorithm based on affinity graph learning.

[0028] Further preferably, the method further includes: the first segmenter uses a nnU-Net network as a model architecture and uses a 3D U-Net as a backbone network for feature extraction.

[0029] Further preferably, the method further comprises, during training, using an improved algorithm based on a mean square error loss function, and the mean square error loss MSE generated under n training data is defined as:

[0030]

[0031] Among them, n represents n training data x i , the true output of each training data is y i , the model is x i The predicted value is

[0032] Further preferably, the method further comprises, when training the first segmenter, using the image segmentation maximization affinity learning loss function MALIS, and combining it with the mean square error loss function MSE, to obtain the loss function to be optimized:

[0033] Loss = w1×MSE+w2×MALIS

[0034] Among them, w1 represents the weight value of the MES, and w2 represents the weight value of the MALIS.

[0035] Further preferably, the value of w1:w2 is 1:1.

[0036] The present invention provides a method for reconstructing electron microscope volume data, which uses a mature nnU-Net electron microscope image segmentation algorithm framework, redesigns the loss function and the output layer of the network, and applies it to the prediction task of affinity graphs. On the one hand, a voxel enhancement algorithm is used to preprocess the volume, and the cycle generative adversarial network (CycleGenerative Adversarial Network, CycleGAN) is improved to obtain a voxel enhancement algorithm (VoxelEnhance GAN, VECGAN) to preprocess the volume data. The method preprocesses the data by synthesizing the volume to reduce the defects in the original volume data with defects. On the other hand, the volume is segmented using a segmentation algorithm based on affinity graph learning, which uses an affinity graph learning method to perform image segmentation on electron microscope data of continuous brain slices, which can effectively improve the expression ability of the model and accurately capture the connection relationship between neuronal cells without relying on clear edge markings. In the method provided by the present invention, the affinity graph is used as a model to provide richer information, allowing the model to establish effective associations between neuronal voxels, so that the model has the ability to predict longer-distance connectivity. Furthermore, the volume data obtained by using the electron microscope volume data reconstruction method provided by the present invention has the characteristics of high reconstruction efficiency and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0038] Figure 1 A flow chart of a method for reconstructing volume data of an electron microscope provided by the present invention;

[0039] Figure 2A network architecture diagram of a voxel enhancement algorithm provided by the present invention;

[0040] Figure 3 (a) is a volume schematic diagram of a lost slice provided by the present invention;

[0041] Figure 3 (b) A voxel enhancement algorithm provided by the present invention is used to Figure 3 (a) Schematic diagram of the volume after processing the volume of the missing slice;

[0042] Figure 4 An example of an affinity diagram provided by the present invention;

[0043] Figure 5 The present invention provides Figure 4 The affinity graph in one direction actually represents;

[0044] Figure 6 This is a graph of affinity graph prediction results after being processed by an image segmentation algorithm based on affinity graph learning provided by the present invention. DETAILED DESCRIPTION

[0045] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0046] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0047] The first and second in the following text are only for distinction and have no other meanings.

[0048] The present invention provides an electron microscope volume data reconstruction method, which can quickly and accurately repair defects in original volume data with defects, and further increase the original distance connectivity between volume data, thereby improving the efficiency of electron microscope volume data reconstruction. The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0049] Figure 1 This is a flow chart of a method for reconstructing volume data of an electron microscope provided by the present invention. As shown in the figure, the method for reconstructing volume data of an electron microscope provided by the present invention comprises the following steps:

[0050] Step 110 , preprocess the original volume Y using a pre-trained voxel enhancement algorithm to obtain first volume data fake-X.

[0051] Specifically, in the embodiment of the present invention, the original volume data Y is volume data with defects. For example, Figure 2 A network architecture diagram of a voxel enhancement algorithm provided by the present invention, wherein the volume Y indicated by Y, and Figure 3 (a) is a volume schematic diagram of a lost slice provided by the present invention, in which the volume of the lost slice is indicated.

[0052] In the preferred embodiment of the present invention, the pre-trained voxel enhancement algorithm (VECGAN) used is an improved cycle generative adversarial network (CycleGAN) algorithm. That is, the pre-trained voxel enhancement algorithm provided by the present invention is an improved algorithm based on the cycle generative adversarial network. The method pre-processes the volume data by synthesizing the volume, which can reduce the defects in the volume data. For example, lost slice data, repair of blurred imaging, correction of lens distortion, repair of slice lossless images and dust particles, etc.

[0053] The voxel enhancement algorithm VECGAN provided by the present invention can learn the transformation between two input volumes and understand the structure and content of the image so as to fill in the missing or incomplete parts of the original image when synthesizing the image, thereby improving the integrity of the data and providing a data basis for subsequent algorithms.

[0054] Step 120, the volume data X and the first volume data fake-X are used as inputs of the pre-trained first segmenter for segmentation processing, and the discriminator E is used to perform discriminant processing on the segmented data, and the data obtained from fake-X is reconstructed into microstructure data.

[0055] Specifically, in the embodiment of the present invention, the first segmenter is pre-trained and has fixed parameters. It is used to segment the input data to obtain first segmented image data. The segmented first segmented image data is then input into the discriminator E for discriminant processing, and the data obtained after segmentation by fake-X is reconstructed and finally reconstructed into microstructure data.

[0056] In a preferred embodiment of the present invention, the first segmenter is an image segmentation algorithm based on affinity graph learning. The affinity graph learning method is used to perform image segmentation on electron microscope data of continuous sections. In the training of the first segmenter, the introduction of the affinity graph improves the expressive power of the segmenter model, and can accurately capture the connection relationship between cells without relying on clear edge markings. In particular, when image segmentation is performed on electron microscope data of continuous sections of the brain, the continuous relationship between neuronal cells can be accurately captured without relying on clear edge markings. In this method, the affinity graph provides the model with richer information, allowing the model to establish effective associations between cell voxels, especially for brain meta-cells, and between neuronal voxels, so that the model has the ability to predict longer-distance connectivity.

[0057] That is to say, in the embodiment scheme provided by the present invention, the image segmentation algorithm based on affinity graph learning is used to train the obtained segmenter, which can more accurately identify and segment neurons when processing image segmentation tasks with complex neural network structures.

[0058] The above describes the process of a method for reconstructing volume data of an electron microscope provided by the present invention. The following describes in detail the network architecture for executing a method for reconstructing volume data of an electron microscope provided by the present invention by way of example.

[0059] Figure 2 This is a network architecture diagram of a voxel enhancement algorithm provided by the present invention. Figure 2 As shown, in the preferred solution of the embodiment of the present invention, the network architecture for pre-training of the Cycle Generative Adversarial Network (CycleGAN) algorithm is improved, mainly including: generator X, generator Y, discriminator X, discriminator Y, segmenter S and discriminator E. Figure 2-6 Each component of the network architecture of the model is described in detail.

[0060] Generator X is used to convert volume data X into volume data fake-Y. In the preferred embodiment provided by the present invention, the backbone network of generator X adopts U-Net neural network. Generator X is used to convert volume data X without defects into volume data fake-Y with defects.

[0061] Generator Y is used to convert volume data Y into first volume data fake-X. In the preferred embodiment provided by the present invention, the backbone network of generator Y adopts U-Net neural network. The function of generator Y is opposite to that of generator X, and is used to convert the original volume Y with defects into the first volume data fake-X without defects.

[0062] It should be noted that the purpose of using opposite generators X and Y in the embodiment of the present invention is to interchange data of different styles.

[0063] Discriminator X and discriminator Y are used to discriminate the authenticity of input data. In another preferred embodiment of the present invention, the backbone network of discriminator X and discriminator Y adopts a multi-layer convolutional neural network. Figure 2 As shown, the discriminator X and the discriminator Y are used to discriminate whether the data they receive are generated by the generator X and the generator Y, respectively. The discriminator X and the generator X form an adversarial relationship, and the discriminator Y and the generator X form an adversarial relationship. During the training process, they are iteratively optimized with each other.

[0064] The segmenter S is used to segment the input data to obtain first segmented image data. The segmenter S in the present invention is pre-trained, has fixed parameters, and is used to segment the volume.

[0065] In a specific example of an embodiment of the present invention, Figure 2 As shown, the segmenter S segments the volume data X and the first volume data fake-X generated by the generator Y, and then sends the two segmentation results to the discriminator E.

[0066] In a preferred solution of the embodiment of the present invention, the segmenter S is an image segmentation algorithm based on affinity graph learning. In a specific example provided by the preferred solution of the embodiment of the present invention, the segmenter S uses the nnU-Net network as the architecture of the model and uses 3D U-Net as the backbone network for feature extraction.

[0067] In a specific embodiment provided by the present invention, the affinity graph is defined such that the value of the edge between nodes in the same instance part is set to 1, while the value of the edge between nodes not in the same instance part is set to 0. Figure 4 An affinity graph example provided by the present invention shows the affinity graph in the form of a graph, wherein the green edges represent the affinity of adjacent pixels in the x-axis direction of the image, and the red edges represent the affinity of adjacent pixels in the y-axis direction. Figure 5 The present invention provides Figure 4 The affinity graph in one direction actually represents .

[0068] In the embodiment provided by the present invention, since nnU-Net was originally used for the semantic segmentation model of medical images, it is not directly used to train the affinity graph prediction model. Therefore, the present invention is improved based on the mean square error loss function (MSE Loss). The specific training method is as follows:

[0069] Assume there are n training data x i , the true output of each training data is y i, the model is x i The predicted value is Then the mean square error loss generated by the model under n training data can be defined as:

[0070]

[0071] In the above formula 1, n represents n training data x i , the true output of each training data is y i , the model is x i The predicted value is

[0072] In order to solve the shortcoming that the mean square error loss function cannot make good use of the connectivity relationship between voxels during training, the preferred solution of the embodiment provided by the present invention introduces the maximum affinity learning of image segmentation (MALIS) loss function. By combining the MSE loss function and the MALIS loss function, the final loss function to be optimized is obtained:

[0073] Loss=w1×MSE+w2×MALIS (Formula 2)

[0074] In the above formula 2, w1 represents the weight of MSE, and w2 represents the weight of MALIS. In a preferred embodiment of the present invention, the ratio of w1 to w2 is 1:1, that is, w1:w2=1:1, and the values ​​of w1 and w2 can be adjusted according to the experimental characteristic data, or during the training process, the ratio can be adjusted according to the experimental data so that the model can achieve the best effect.

[0075] In the preferred scheme provided by the embodiment of the present invention, after implementing the training of the image segmentation algorithm based on affinity graph learning, the inventor tested the trained segmentation algorithm using a public data set, and the test results were as follows: the variation of information split (VOI split) was 0.472, and the variation of information merge (VOI merge) was 0.074.

[0076] Figure 6 The affinity graph prediction result diagram after being processed by an image segmentation algorithm based on affinity graph learning provided by the present invention is shown in the figure. A specific example in an embodiment of the present invention is shown, which is a prediction result diagram after being processed by an image segmentation algorithm based on affinity graph learning provided by the present invention.

[0077] Furthermore, the inventors compared the trained segmentation algorithm with several other popular image segmentation models, and the experimental results are shown in Table 1. By comparing with other models, the experimental results on the Circuit Reconstruction from Electron Microscopy Images (CREMI) dataset show that the proposed method performs well in the Variation Of Information (VOI) indicator. This shows that the improved nnU-Net does a good job in avoiding over-segmentation (Variation Of Information split, VOI split) and under-segmentation (Variation Of Information merge, VOI merge), and the total VOI value is relatively low, which means that the difference between the segmentation result and the true segmentation result is small.

[0078] Table 1 Comparison of experimental results on the CREMI dataset

[0079]

[0080] The discriminator E is used to discriminate the source of the first segmented image data. In a preferred embodiment of the present invention, the discriminator E is the same as the discriminator X, and its backbone network adopts a multi-layer convolutional neural network. In another preferred embodiment of the present invention, the backbone network of the discriminator E adopts a multi-layer convolutional neural network. The backbone network type of the discriminator can be selected according to the type of slice actually reconstructed.

[0081] In a specific example provided by the embodiment of the present invention, Figure 2 As shown, the discriminator E is used to discriminate whether the input segmented data is generated by the volume data X or the first volume data fake-X is generated after segmentation by the segmentor.

[0082] In the preferred embodiment of the present invention, a cyclic generative adversarial network is used as the backbone network. When training the voxel enhancement algorithm, G is used to represent the generator, D is used to represent the discriminator, and S is used to represent the segmenter. When training the generator Y, the loss function to be optimized is:

[0083] |G x (G Y (Y))-Y|+(D X (G Y (Y))-1) 2 +(D S (S(G Y (Y)))-1) 2 (Formula 3)

[0084] The loss function that needs to be optimized when training the discriminator E is:

[0085] (D s (S(G Y (Y)))-0) 2 +(D S (S(X))-1) 2 (Formula 4)

[0086] In the above equations 3 and 4, X represents volume data X, Y represents original volume Y, G represents generator, D represents discriminator, and S represents segmenter.

[0087] The above describes the network architecture of the electron microscope volume data reconstruction method provided by the present invention. Figure 2 The network architecture of the mat microscope volume data reconstruction method using this architecture is introduced. Figure 2 This is a network architecture diagram of a voxel enhancement algorithm provided by the present invention. As shown in the figure, the volume data X with defects is input into the generator X and converted into the volume data fake-Y; then the volume data Y with defects is input into the generator Y and converted into the first volume data fake-X, and then the discriminant X is processed. Therefore, after iterative training, the first volume data fake-X output by the generator Y and the volume data X are input into the segmenter S for segmentation processing, and the segmented data is used as the input of the discriminator E. The discriminator E determines the source of the input segmented data, and reconstructs the segmented data obtained by segmenting fake-X by the segmenter S into microstructure volume data.

[0088] The present invention provides a method for reconstructing electron microscope volume data, which uses a mature nnU-Net electron microscope image segmentation algorithm framework, redesigns the loss function and the output layer of the network, and applies it to the prediction task of affinity graphs. On the one hand, a voxel enhancement algorithm is used to preprocess the volume, and the cycle generative adversarial network (CycleGenerative Adversarial Network, CycleGAN) is improved to obtain a voxel enhancement algorithm (VoxelEnhance GAN, VECGAN) to preprocess the volume data. The method preprocesses the data by synthesizing the volume to reduce the defects in the original volume data with defects. On the other hand, the volume is segmented using a segmentation algorithm based on affinity graph learning, which uses an affinity graph learning method to perform image segmentation on electron microscope data of continuous brain slices, which can effectively improve the expression ability of the model and accurately capture the connection relationship between neuronal cells without relying on clear edge markings. In the method provided by the present invention, the affinity graph is used as a model to provide richer information, allowing the model to establish effective associations between neuronal voxels, so that the model has the ability to predict longer-distance connectivity. Furthermore, the volume data obtained by using the electron microscope volume data reconstruction method provided by the present invention has the characteristics of high reconstruction efficiency and high accuracy.

[0089] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0090] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0091] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for reconstructing volume data of an electron microscope, characterized in that: The method comprises: Preprocess the original volume Y using a pre-trained voxel enhancement algorithm to obtain the first volume data fake-X; The volume data X and the first volume data fake-X are used as inputs of a pre-trained first segmenter to perform segmentation processing, and a discriminator E is used to perform discriminant processing on the segmented data to reconstruct the data obtained from fake-X into microstructure data; The pre-trained voxel enhancement algorithm is an improved cyclic generative adversarial network algorithm, and the network architecture of the improved cyclic generative adversarial network algorithm includes: A generator X, configured to convert the volume data X into volume data fake-Y; A generator Y, configured to convert the volume data Y into the first volume data fake-X; Discriminator X and discriminator Y are used to determine whether the input data is true or false; A segmenter S, used for segmenting the input data to obtain first segmented image data; A discriminator E, used for discriminating the source of the first segmented image data; and The loss function when training the generator Y is: |G X (G Y (Y))-Y|+(D X (G Y (Y))-1) 2 +(D S (S(G Y (Y)))-1) 2 The loss function when training the discriminator E is: (D S (S(G Y (Y)))-0) 2 +(D S (S(X))-1) 2 Wherein, X represents the volume data X, Y represents the original volume Y, G represents the generator, D represents the discriminator, and S represents the segmenter; and, The first segmenter uses the nnU-Net network as the architecture of the model and uses 3D U-Net as the backbone network for feature extraction; and During training, an improved algorithm based on the mean square error loss function is used, and the mean square error loss MSE generated under n training data is defined as: Among them, n represents n training data x i , the true output of each training data is y i , the model is x i The predicted value is When training the first segmenter, the image segmentation maximization affinity learning loss function MALIS is used, and combined with the mean square error loss function MSE, the loss function to be optimized is obtained as follows: Loss = w1×MSE+w2×MALIS Wherein, w1 represents the weight value of the MSE, and w2 represents the weight value of the MALIS.

2. The method according to claim 1, characterized in that: The backbone networks of the generator X and the generator Y adopt U-Net neural network.

3. The method according to claim 1, characterized in that: The backbone networks of the discriminator X, the discriminator Y and the discriminator E adopt a multi-layer convolutional neural network.

4. The method according to claim 1, characterized in that: The method also includes: the first segmenter is an image segmentation algorithm based on affinity graph learning.

Images