Intelligent interpolation method and system for three-dimensional microscopic images of fibrous structures

Abstract

The present invention provides an intelligent interpolation method and system for three-dimensional microscopic images of fibrous structures. First, a zy surface image similar to anisotropic low-resolution is constructed as a training data set. Secondly, a full convolutional neural network model is trained with a training data set constructed by an xy surface. Then, the trained model is applied to the zy surface low-resolution data. After prediction by the full convolutional neural network model, the resolution in the z direction is improved. The image result predicted by the present invention is significantly improved compared with the original zy surface low-resolution image quality, and the sawtooth phenomenon on the original low-resolution image is improved to a great extent. The predicted image neuron fibers are clearer; at the same time, the grayscale curve graph drawn by the zy surface image is compared, and the curve of the predicted image is more continuous and smooth, eliminating the sawtooth effect of the original z-direction low-resolution image; the predicted zy surface image is re-sliced ​​to achieve high-resolution isotropy of the three-dimensional data set.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an intelligent interpolation method and system for fibrous three-dimensional structural information of neurons, blood vessels, etc. in a brain microscopic image at a single-cell resolution level. Background Art

[0002] One of the research directions of modern neuroscience is the analysis of complex brain structures. In recent years, with the rapid development of neuron sparse labeling technology and whole-brain optical imaging technology at the mesoscopic scale, technical conditions have been provided for neurologists to study the neural circuits of the brain at the mesoscopic level. To study neural circuits, it is necessary to image the whole brain, and the observation accuracy must reach the micrometer level. At the macroscopic scale, imaging technologies such as MRI can observe obvious spatial features such as brain regions, but the observation accuracy is not enough and can only be at the millimeter level. The spatial resolution of whole-brain imaging at the centimeter scale can only reach 10-50 μm at most; at the microscopic scale, the fine morphology of neurons can be observed through electron microscopes, and its resolution can even reach 1nm. Due to the limitations of imaging time and cost, electron microscopes can only scan very small volumes of images, so they can only image local branches of neurons; at the mesoscopic scale, between the two, it helps to reconstruct the real brain neural network and is an important part of brain science research. In particular, optical microscopy can detect the fine structure of biological tissues at the single-cell resolution level. Using a sophisticated microscopic optical imaging platform, the thickness of the cut tissue slices can reach 1 μm, which has the ability to obtain a three-dimensional data set at the resolution level of a single neuron in the whole mouse brain. However, due to the physical limitations of the optical imaging system, the images collected by the microscopic optical imaging system are anisotropic, which affects the subsequent visualization of the three-dimensional image data set and image-based data analysis, such as high-resolution image registration, neuron radius estimation, and neuron reconstruction.

[0003] Therefore, there is an urgent need for a fast, automatic, high-resolution, isotropic, intelligent interpolation method to improve the resolution of massive image data sets, such as disclosed in China's public patents CN105590338A "A three-dimensional reconstruction method for scanning electron microscopy images" and CN104751425A "A fluorescence microscopy image reconstruction method based on spatially varying point spread function". However, due to the influence of the point spread function of the optical imaging system, the three-dimensional voxel resolution xyz of the acquired image has a z-direction resolution that is much lower than the xy-direction resolution, and the anisotropy of the three-dimensional data set is difficult to overcome at the physical device level. Therefore, using image processing algorithms to improve the z-direction resolution of the image acquired by the optical system and achieve isotropy of the three-dimensional data set has become a new idea to solve this problem.

[0004] At present, for conventional 3D data sets collected, there is a problem of relatively low resolution in a certain direction. Image interpolation technology can be used to compensate for the low resolution problem in this direction. Image interpolation is a process of regenerating image data, which uses low-resolution original image data to regenerate higher-resolution image data. The goal of sequence image interpolation is to generate a new interpolated image between two adjacent images through interpolation methods, which increases the number of images between the sequence and reduces the spacing between images, which is helpful for subsequent image 3D reconstruction. Generally speaking, image interpolation can be divided into two categories: one is reconstruction-based methods and the other is learning-based methods. Reconstruction-based methods use prior information to formulate interpolation functions, such as Markov models, autoregressive models, and non-local similarity. However, these reconstruction-based methods rely on the fact that the observed low-resolution image information is very limited, often resulting in blurred edges of the image. Learning-based methods use a lot of information from training samples, such as sparse representation, random forests, deep neural networks, etc. The key to learning-based methods is to extract relevant information from training samples to interpolate and generate new pixels. However, if deep learning methods are directly applied to the problem of restoring the isotropy of optical microscope images, the problem is complicated by the lack of sufficient training data and the impossibility of manually generating such data, resulting in the inability of deep learning methods to be directly used for the high-resolution isotropic interpolation problem of mesoscopic-scale datasets. Summary of the invention

[0005] In view of the shortcomings of the current existing technology, the purpose of the present invention is to provide an intelligent interpolation method and system for three-dimensional microscopic images of fibrous structures, so as to improve the resolution in the z direction (axial direction) of the three-dimensional image xyz with a single-cell resolution level, and to achieve isotropy of the spatial resolution of the three-dimensional microscopic image on the basis of maintaining the resolution in all directions at the same order of magnitude.

[0006] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0007] An intelligent interpolation method for a three-dimensional microscopic image of a fibrous structure, the method comprising the following steps:

[0008] Step S1, a step of obtaining raw data, obtaining a three-dimensional image dataset of brain tissue at a single-cell resolution level;

[0009] Step S2, a training data set construction step, cutting an xy plane high-resolution two-dimensional data set A from the three-dimensional image data set to construct a gold standard data set for training a full convolutional network model; downsampling the xy plane high-resolution two-dimensional data set A to generate a low-resolution two-dimensional data set B;

[0010] Step S3, a fully convolutional network model training step, inputting the two-dimensional data sets A and B into the fully convolutional network model for training until the model can perform low-resolution to high-resolution prediction of two-dimensional images;

[0011] Step S4, a high-resolution image prediction step in the z direction, inputting the low-resolution two-dimensional image data of the zy surface into the trained fully convolutional network model for prediction, and generating corresponding high-resolution two-dimensional image data of the zy surface;

[0012] Step S5, a loop processing step, loop processing all the zy plane and zx plane low-resolution two-dimensional image data of the three-dimensional image data set to obtain all the zy plane and zx plane high-resolution two-dimensional image data;

[0013] Step S6, a three-dimensional high-resolution result output step, re-slices and restores the high-resolution two-dimensional image data of the zy plane and the zx plane, and outputs a high-resolution three-dimensional image data set of brain tissue at the single-cell resolution level.

[0014] Preferably, the two-dimensional image data of the zy plane and the zx plane are all obtained by re-slicing the three-dimensional image data set.

[0015] Preferably, the "cropped xy plane high-resolution two-dimensional dataset A" in step S2 is specifically a continuous slice image in the xy direction of the three-dimensional image of the brain tissue, and the positions with signal points in the image are selected and cropped with a size of 128 x 128 pixels as the gold standard dataset for training the full convolutional network model.

[0016] Preferably, the "downsampling of the xy plane high-resolution two-dimensional dataset A" in step S2 is specifically to downsample the xy plane image along the x or y direction through a bicubic interpolation method to generate a low-resolution two-dimensional dataset B, thereby simulating the high-resolution two-dimensional image data of the zy plane or zx plane, thereby constructing a high-resolution and low-resolution matching training dataset.

[0017] Preferably, the full convolutional network model in step S3 is composed of a convolutional layer, a maximum pooling layer and an upsampling layer, so that it can extract information features of the image and predict a high-resolution image; by adjusting the parameters of the full convolutional network structure, the network model under the optimal parameters in the experimental test is selected, and the parameter optimization is to design the number of convolutions, maximum pooling and upsampling of the network while ensuring the accuracy and similarity of the predicted image and taking reducing the time of network training and prediction as the criterion.

[0018] Preferably, the step S3 of “inputting the two-dimensional data sets A and B into a fully convolutional network model for training” specifically includes:

[0019] Input a low-resolution two-dimensional dataset B, first generate a dataset B' through image preprocessing and upsampling, so that the dataset B' has the same size as the dataset A, ensuring that the input and output images of the full convolutional network model have the same size, and at the same time correspond one-to-one with the gold standard dataset A, use the full convolutional network model to learn the transformation from low-resolution data B' to high-resolution data A, and after repeated training, select the network model weight result with the best predicted image.

[0020] Preferably, the step S4 specifically includes:

[0021] Step S41, split a single zy surface low-resolution two-dimensional image into a series of small images; the reason for splitting into a series of small images is that the resolution of a single zy surface image is tens of thousands of pixels, which cannot be directly input into the full convolutional network model for prediction, where the zy surface image is obtained by re-slicing the three-dimensional image data. Therefore, it is necessary to first sample a complete zy surface image along the x or y direction to be isotropic, ensure that the pixels of z and x or z and y are consistent, generate a new low-resolution image zy', and then split the newly generated low-resolution image into smaller images, and input them into the full convolutional network for prediction;

[0022] Step S42: input each small image into the trained full convolutional network model in sequence, perform pixel-by-pixel prediction, and obtain high-resolution isotropic two-dimensional image data of each small image;

[0023] Step S43, stitching the predicted small images with high resolution into a complete large image to obtain a high-resolution isotropic zy surface image.

[0024] Preferably, the step S43 of "stitching the predicted small images with high resolution into a complete large image" specifically comprises splitting each high-resolution isotropic small image obtained after prediction in step S42 into small images in the order of splitting the large image, setting the redundancy value of the small image boundary during stitching, and re-stitching and restoring them into a complete zy surface large image, thereby obtaining a high-resolution isotropic zy surface image.

[0025] The present invention also discloses an intelligent interpolation system for three-dimensional microscopic images of fibrous structures, comprising:

[0026] Acquiring raw data units for acquiring a three-dimensional image dataset of brain tissue at a single-cell resolution level;

[0027] Constructing a training data set unit, for cutting an xy plane high-resolution two-dimensional data set A from the three-dimensional image data set to construct a gold standard data set for training a full convolutional network model; downsampling the xy plane high-resolution two-dimensional data set A to generate a low-resolution two-dimensional data set B;

[0028] A full convolutional network model training unit, used for inputting the two-dimensional data sets A and B into the full convolutional network model for training until the model can perform low-resolution to high-resolution prediction of the two-dimensional image;

[0029] A z-axis high-resolution image prediction unit is used to input the zy-plane low-resolution two-dimensional image data into the trained full convolutional network model for prediction, and generate the corresponding zy-plane high-resolution two-dimensional image data;

[0030] a loop processing unit, used for loop processing all the zy plane and zx plane low-resolution two-dimensional image data of the three-dimensional image data set to obtain all the zy plane and zx plane high-resolution two-dimensional image data;

[0031] The three-dimensional high-resolution result output unit is used to re-slice and restore the high-resolution two-dimensional image data of the zy plane and zx plane, and output a three-dimensional image data set of brain tissue with a high-resolution single-cell resolution level.

[0032] Preferably, the full convolutional network model is composed of a convolutional layer, a maximum pooling layer, and an upsampling layer, so that it can extract information features of the image and predict a high-resolution image; by adjusting the parameters of the full convolutional network structure, the network model under the optimal parameters in the experimental test is selected, and the parameter optimization is to design the number of convolutions, maximum pooling, and upsampling of the network while ensuring the accuracy and similarity of the predicted image and taking reducing the time of network training and prediction as the criterion.

[0033] Preferably, the “inputting the two-dimensional data sets A and B into the fully convolutional network model for training” in the fully convolutional network model training unit is specifically,

[0034] Input a low-resolution two-dimensional dataset B, first generate a dataset B' through image preprocessing and upsampling, so that the dataset B' has the same size as the dataset A, ensuring that the input and output images of the full convolutional network model have the same size, and at the same time correspond one-to-one with the gold standard dataset A, use the full convolutional network model to learn the transformation from low-resolution data B' to high-resolution data A, and after repeated training, select the network model weight result with the best predicted image.

[0035] Preferably, the z-direction high-resolution image prediction unit splits a single zy surface low-resolution two-dimensional image into a series of small images; each small image is sequentially input into a trained fully convolutional network model for prediction to obtain high-resolution two-dimensional image data of each small image; each high-resolution isotropic small image obtained after prediction is split into small images in the order of the large image, and the redundancy value of the small image boundaries during splicing is set, and the images are re-spliced ​​to restore them into a complete zy surface large image, thereby obtaining a high-resolution isotropic zy surface image.

[0036] The beneficial effects of the present invention are mainly reflected in: the method of the present invention has a significant improvement effect on the resolution of the fibrous structure three-dimensional image data set, and realizes the high-resolution isotropy of the three-dimensional data set. The low-resolution anisotropic image similar to the zy surface constructed on the xy surface high-resolution isotropic data set is predicted, and the predicted image obtained is qualitatively compared with the original gold standard image. The human eye observation is very similar to the gold standard image. The PSNR mean of the quantitative analysis is more than 80, and the SSIM mean is more than 88%; at the same time, the grayscale curve change of the predicted image is highly consistent with the gold standard image; the training data set constructed by the xy surface high-resolution image is then used. The training result of the full convolution network model is applied to the zy surface low-resolution data set, and the predicted image result is significantly improved compared to the original zy surface low-resolution image quality, which greatly improves the sawtooth phenomenon on the original low-resolution image, and the predicted image neuron fiber is clearer. At the same time, the grayscale curve graph drawn by the zy surface image is compared, and the curve of the predicted image is more continuous and smooth, eliminating the sawtooth effect of the original z-direction low-resolution image. The predicted zygomatic image is re-sliced ​​and restored to the original three-dimensional image direction, thereby achieving high-resolution isotropy of the three-dimensional data set.

[0037] The method of the present invention is suitable for improving the resolution of a three-dimensional image data set collected by a microscopic optical imaging system, thereby achieving high-resolution isotropy of the three-dimensional image data set, which is beneficial to subsequent data visualization, neuron radius estimation, neuron reconstruction and other applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic flow chart of an intelligent interpolation method for a three-dimensional microscopic image of a fibrous structure according to an embodiment of the present invention;

[0040] Figure 2 The xy-plane image and the two-dimensional image of the zy-plane after re-sectioning were imaged using the fluorescence microscopy optical sectioning tomography system (fMOST).

[0041] Figure 3 A schematic diagram of the fully convolutional network structure established for the present invention;

[0042] Figure 4 Comparison chart of high-resolution isotropic prediction results in the z direction of a three-dimensional dataset. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] See also Figure 1 As shown, one embodiment of the present invention discloses an intelligent interpolation method for a three-dimensional microscopic image of a fibrous structure, the method comprising the following steps:

[0045] Step S1, a step of obtaining raw data, obtaining a three-dimensional image dataset of brain tissue at a single-cell resolution level;

[0046] Step S2, a training data set construction step, cutting an xy plane high-resolution two-dimensional data set A from the three-dimensional image data set to construct a gold standard data set for training a full convolutional network model; downsampling the xy plane high-resolution two-dimensional data set A to generate a low-resolution two-dimensional data set B;

[0047] Step S3, a fully convolutional network model training step, inputting the two-dimensional data sets A and B into the fully convolutional network model for training until the model can perform low-resolution to high-resolution prediction of two-dimensional images;

[0048] Step S4, a high-resolution image prediction step in the z direction, inputting the low-resolution two-dimensional image data of the zy surface into the trained fully convolutional network model for prediction, and generating corresponding high-resolution two-dimensional image data of the zy surface;

[0049] Step S5, a loop processing step, loop processing all the zy plane and zx plane low-resolution two-dimensional image data of the three-dimensional image data set to obtain all the zy plane and zx plane high-resolution two-dimensional image data;

[0050] Step S6, a three-dimensional high-resolution result output step, re-slices and restores the high-resolution two-dimensional image data of the zy plane and the zx plane, and outputs a high-resolution three-dimensional image data set of brain tissue at the single-cell resolution level.

[0051] Next, a mouse brain sample is used as an example. The mouse brain sample used in the method of the present invention is labeled with a virus carrying yellow fluorescent protein (EYFP) and imaged using a fluorescence microscopy optical sectioning tomography system (fMOST). Figure 2As shown, the xy plane is a high-resolution image, and the zy plane is a low-resolution image. The image resolution is X 0.3μm × Y 0.3μm × Z 1μm, and the axial (Z-direction) resolution is only about one-third of the lateral resolution. Therefore, the present invention takes the image of the neuronal fiber channel in the collected image as the research object, improves the axial (Z-direction) resolution of the image, and thus realizes the high-resolution isotropy of the three-dimensional data set.

[0052] 1. Training Dataset Generation Module

[0053] For microscopic optical section tomography systems, the xy-plane resolution of the captured images is high, but the z-plane resolution is low, and it is difficult to obtain a real data set with high resolution in the z-plane as a gold standard for training. If multiple images are to be interpolated between two consecutive xy-plane images, it is difficult and there is no real data set for comparison and learning, so it is impossible to use the existing image interpolation method for reconstruction.

[0054] The mechanism of the present invention is to convert a three-dimensional data set into a two-dimensional image of the zy plane, and then predict the low-resolution image data of the zy plane to obtain high-resolution image data in the z direction. The full convolutional network model based on deep learning requires a large amount of training data sets for training. In order to generate the training data set, there are the following two steps, including:

[0055] 1. Crop the xy plane high-resolution dataset A module

[0056] Since the neuronal fiber information of the xy plane image and the zy plane image is highly similar, the present invention selects to crop continuous xy plane images as the gold standard images of the training data set. The cropped images are selected as images containing neuronal fiber signals as much as possible. The cropped image size is 128 x 128 pixels, and it is necessary to crop about 10,000 to 20,000 small images as the training data set.

[0057] 2. Downsample dataset A to generate dataset B, simulating the low-resolution data module in the z direction

[0058] The cropped xy-plane high-resolution image dataset is used as the gold standard image for training, and low-resolution data similar to the z-direction is constructed through computational simulation to generate high-resolution and low-resolution matching training image pairs. The specific explanation is as follows: According to the three-dimensional voxel resolution of the mouse brain image collected by the microscope optical system, the xyz is 0.3 μm × 0.3 μm × 1 μm, and the resolution of the zy plane obtained by re-slicing is 1 μm × 0.3 μm, and the resolution in the y direction is 3.33 times that in the z direction. Therefore, the image data of the xy plane is downsampled 3.33 times along the x direction using the bicubic interpolation method to simulate low-resolution data similar to the zy plane, thereby generating matching low-resolution image data pairs using the xy-plane high-resolution data.

[0059] 2. Fully Convolutional Network Model Training Module

[0060] The constructed training dataset is trained using the established fully convolutional network model.

[0061] 1. Establish a fully convolutional network model and perform parameter optimization module

[0062] In order to effectively extract the features of low-resolution images to high-resolution images, and make the designed full convolutional network model learn the interpolation process of low-resolution to high-resolution images more effectively, the network structure is designed mainly with convolution kernel, maximum pooling layer and upsampling layer, and the accuracy and similarity of the predicted image are guaranteed. Under the premise of the same predicted image results, the image data set can be trained and predicted faster. Through multiple parameter adjustments and experimental tests, different maximum pooling times, convolution layers, and upsampling times are designed, and the predicted images are compared. The final constructed full convolutional network structure contains three maximum pooling and three upsampling, and each pooling and upsampling contains 2 layers of image convolution layer operations. The whole network structure is divided into two parts, one is the feature extraction part, which includes three maximum pooling layers and 6 convolution layers, and each pooling layer has two convolution operations before and after; the other is the upsampling part. Each time the image is upsampled, the image size becomes larger accordingly, and the image with the same size of the corresponding number of channels in the maximum pooling layer is fused, that is, spliced ​​with each other. Through the upsampling layer, the output image of the network model is kept the same size as the input image.

[0063] To predict high-resolution images through a fully convolutional network, it is necessary to regress each pixel value, so the activation function sigmoid in traditional classification problems cannot be used. The network activation function selected here is the linear parameter rectifier unit PReLU (Parametric ReLU), and its function expression is:

[0064] The activation function PReLU can avoid the phenomenon that some neurons in the neural network model cannot be activated. That is to say, if the neuron input is a non-positive value, the neuron can also be activated, ensuring that each neuron can obtain a corresponding gradient to update the corresponding parameters. The established full convolutional network structure is as follows Figure 3 shown.

[0065] 2. Input the constructed training data sets A and B into the training module in the fully convolutional network

[0066] The high-resolution image dataset A of the xy plane is used as the gold standard for network model training. The image dataset B similar to the zy plane is constructed through computational simulation. Before inputting into the network training, data preprocessing is required. The x and y resolutions of dataset B are upsampled to the same size using bicubic interpolation as the input low-resolution image B' of the network model. The established fully convolutional network model is then used to train the constructed high-resolution and low-resolution matching datasets. After multiple trainings, the best training weight result is selected.

[0067] The computing platform used in the present invention is Windows 10, Intel Xeon CPU E5-2699, the processor is 2.20GHz, the memory is 384GB, and the established full convolutional network is implemented based on Keras.

[0068] 3. Z-axis high-resolution image prediction module

[0069] The low-resolution image dataset in the z direction is predicted through a fully convolutional network to generate a predicted high-resolution image in the z direction. There are three steps, including:

[0070] Step 1, splitting a single zy surface low-resolution two-dimensional image into a series of small images;

[0071] For the three-dimensional data set of mouse brain collected by the microscopic optical sectioning tomography system, a single xy plane image has tens of thousands of pixels, and the zy plane image also has tens of thousands of pixels. The zy plane is an image obtained by re-slicing the three-dimensional data set xyz. Therefore, when predicting the full convolutional network, such a large image cannot be directly input. Therefore, it is necessary to split the complete zy plane image into a series of small images, and then input them into the network for prediction. Here, a complete original resolution zy plane image obtained by re-slicing is first upsampled to isotropic resolution by bicubic interpolation, and used as the input low-resolution image of the network. Then, a single zy plane image is cropped. You can choose to crop it into a small image of 1000 x 1000 pixels. At the same time, the redundancy (RD, Redundance) parameter will be set at the edge of the image during cropping to ensure that no obvious traces will be produced at the edge of the image during splicing.

[0072] Step 2: Input each small image into the trained full convolutional network model in turn to perform pixel-by-pixel prediction to obtain high-resolution two-dimensional image data of each small image;

[0073] The network weights trained in step 1 are applied to the prediction of the zy face image. Each 1000 x 1000 pixel image split from a single zy face image is input into the network for prediction. The prediction is performed with a stride of 100 and a sliding window (Img_size) of 128 x128 pixels. At the same time, it is ensured that there is a certain redundancy (RD, Redundance) area between each small adjacent sliding window. The expression is:

[0074] Here n represents the number of overlapping areas, and the average value of the redundant areas is calculated as the final result to ensure that there will be no obvious seams when splicing a single small image.

[0075] The image splitting in the present invention is to set a pixel size to crop the image, such as 1000x1000 pixel in the embodiment. It only needs to set the size xy pixels to be the same and larger than 128x128 pixel. Generally, it is better to choose a multiple of 100 for the convenience of calculation.

[0076] Step 3: stitch the predicted small images with high resolution into a complete large image to obtain a high-resolution isotropic zy surface image;

[0077] Each 1000 x 1000 pixel predicted in step 2 is stitched and restored into a complete zy face image according to certain redundant parameters. By stitching a series of small images after prediction, a high-resolution isotropic image can be obtained. The predicted zy face image is then restored to the original three-dimensional image through data re-slicing. The resolution of the obtained three-dimensional data set is isotropic, thereby improving the resolution of the three-dimensional data set and obtaining a high-resolution isotropic three-dimensional data set.

[0078] like Figure 4 The three-dimensional data set is shown in the figure. The first row is the comparison between the xy plane image restored by re-slicing the predicted image and the xy plane image of the original data. The second row is the comparison between the zy plane image prediction and the zy plane result image of the original data re-slicing. It can be seen that the image quality has been significantly improved, and the jagged phenomenon on the original low-resolution image has been greatly improved. The predicted image neuron fibers are clearer.

[0079] The third row is a comparison of the grayscale curve change of the orange line in the second row of images. It can be seen that the curve of the predicted image is more continuous and smooth, eliminating the jagged effect of the original low-resolution image in the z direction.

[0080] In the process of constructing a training data set, the present invention converts the interpolation problem of multiple images between layers of a three-dimensional image into a high-resolution prediction of the zy surface of a two-dimensional image, so as to solve the problem that it is difficult for an optical microscope to obtain a high-resolution image in the z direction, resulting in a deep learning method that cannot be directly used for the high-resolution isotropy problem of a mesoscopic three-dimensional data set due to the lack of a gold standard data set in the z direction. At the same time, in order to deal with the problem of a large amount of high-resolution data of three-dimensional images, a single image is split into multiple small images, which are then input into the network for prediction and then spliced ​​and restored into a large image. In addition, the high-resolution isotropic data set in the z direction obtained after the prediction can significantly improve the quality of the image and achieve high-resolution isotropy of the three-dimensional data set compared with the traditional interpolation method.

[0081] The present invention provides an intelligent interpolation method for improving the resolution of a fibrous structure three-dimensional image data set:

[0082] 1. The present invention is based on the fact that different sections of a three-dimensional image dataset have similar image information. The present invention utilizes a high-resolution dataset in the xy plane and constructs data similar to low-resolution features in the zy plane through a computational simulation method to generate a dataset that matches high-resolution and low-resolution data. This solves the problem that deep learning methods cannot be directly used for high-resolution isotropy of mesoscopic-scale datasets due to the lack of a high-resolution gold standard training dataset in the z direction in actual processes.

[0083] 2. Aiming at the problem of processing big data, the present invention first splits the input image into multiple small images, and then performs network prediction and stitches them together to restore them into a complete large image, thus solving the problem that the data volume of three-dimensional microscopic images is huge (on the order of 10GB) and it is difficult to read them into memory for calculation at one time.

[0084] 3. The method of the present invention has wide adaptability, high robustness, good parallelism, is suitable for three-dimensional image data sets of optical microscope imaging, and can achieve high-resolution isotropy of three-dimensional data sets.

[0085] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An intelligent interpolation method for three-dimensional microscopic images of fibrous structures, characterized in that: The method comprises the following steps: Step S1, a step of obtaining raw data, obtaining a three-dimensional image dataset of brain tissue at a single-cell resolution level; Step S2, a training data set construction step, cutting an xy plane high-resolution two-dimensional data set A from the three-dimensional image data set to construct a gold standard data set for training a full convolutional network model; Downsampling the xy plane high-resolution two-dimensional dataset A to generate a low-resolution two-dimensional dataset B; Step S3, a fully convolutional network model training step, inputting the two-dimensional data sets A and B into the fully convolutional network model for training until the model can perform low-resolution to high-resolution prediction of two-dimensional images; Step S4, a high-resolution image prediction step in the z direction, inputting the low-resolution two-dimensional image data of the zy surface into the trained fully convolutional network model for prediction, and generating corresponding high-resolution two-dimensional image data of the zy surface; Step S5, a loop processing step, loop processing all the zy plane and zx plane low-resolution two-dimensional image data of the three-dimensional image data set to obtain all the zy plane and zx plane high-resolution two-dimensional image data; Step S6, a three-dimensional high-resolution result output step, re-slicing and restoring the high-resolution two-dimensional image data of the zy plane and the zx plane, and outputting a high-resolution three-dimensional image data set of brain tissue at a single-cell resolution level; The step S3 of "inputting the two-dimensional data sets A and B into the fully convolutional network model for training" specifically includes: Input a low-resolution two-dimensional data set B, first generate a data set B' through image preprocessing and upsampling, so that the data set B' has the same size as the data set A, ensuring that the input and output images of the full convolutional network model have the same size, and at the same time correspond one-to-one with the gold standard data set A, use the full convolutional network model to learn the transformation from low-resolution data B' to high-resolution data A, and after repeated training, select the network model weight result with the best prediction image; The step S4 specifically includes Step S41, splitting a single zy plane low-resolution two-dimensional image into a series of small images according to a set pixel size; Step S42: input each small image into the trained full convolutional network model in sequence to perform pixel-by-pixel prediction, so as to obtain high-resolution isotropic two-dimensional image data of each small image; Step S43, stitching the predicted high-resolution small images into a complete large image to obtain a high-resolution isotropic zy surface image.

2. The method according to claim 1, characterized in that: The two-dimensional image data of the zy plane and the zx plane are obtained by re-slicing the three-dimensional image data set.

3. The method according to claim 1, characterized in that The "cropped xy plane high-resolution two-dimensional dataset A" in step S2 is specifically a continuous slice image in the xy direction of the three-dimensional image of the brain tissue, where positions with signal points in the image are selected and cropped with a set pixel size as the gold standard dataset for training the full convolutional network model.

4. The method according to claim 1, characterized in that: The "downsampling of the xy plane high-resolution two-dimensional dataset A" in step S2 is specifically to downsample the xy plane image along the x or y direction through a bicubic interpolation method to generate a low-resolution two-dimensional dataset B, thereby simulating the high-resolution two-dimensional image data of the zy plane or zx plane, thereby constructing a high-resolution and low-resolution matching training dataset.

5. The method according to claim 1, characterized in that The full convolutional network model in step S3 is composed of a convolutional layer, a maximum pooling layer, and an upsampling layer, which can extract information features of the image and predict a high-resolution image. By adjusting the parameters of the full convolutional network structure, the network model under the optimal parameters in the experimental test is selected. The optimal parameters are based on the principle of reducing the time of network training and prediction while ensuring the accuracy and similarity of the predicted image.

6. The method according to claim 1, characterized in that The "stitching the predicted high-resolution small images into a complete large image" in step S43 specifically involves splitting each high-resolution small image obtained after prediction in step S42 into small images in the order of splitting the large image, and setting the redundancy value of the small image boundaries during stitching, and re-stitching and restoring them into a complete zy surface large image, thereby obtaining a high-resolution isotropic zy surface image.

7. An intelligent interpolation system for three-dimensional microscopic images of fibrous structures, characterized in that: include: Acquiring raw data units for acquiring a three-dimensional image dataset of brain tissue at a single-cell resolution level; Constructing a training data set unit, for cutting an xy plane high-resolution two-dimensional data set A from the three-dimensional image data set to construct a gold standard data set for training a full convolutional network model; Downsampling the xy plane high-resolution two-dimensional dataset A to generate a low-resolution two-dimensional dataset B; A full convolutional network model training unit, used for inputting the two-dimensional data sets A and B into the full convolutional network model for training until the model can perform low-resolution to high-resolution prediction of the two-dimensional image; The “inputting the two-dimensional data sets A and B into the fully convolutional network model for training” specifically includes: Input a low-resolution two-dimensional data set B, first generate a data set B' through image preprocessing and upsampling, so that the data set B' has the same size as the data set A, ensuring that the input and output images of the full convolutional network model have the same size, and at the same time correspond one-to-one with the gold standard data set A, use the full convolutional network model to learn the transformation from low-resolution data B' to high-resolution data A, and after repeated training, select the network model weight result with the best prediction image; The z-axis high-resolution image prediction unit is used to input the zy-plane low-resolution two-dimensional image data into the trained full convolutional network model for prediction, and generate the corresponding zy-plane high-resolution two-dimensional image data, which specifically includes the following steps: Step S41, splitting a single zy plane low-resolution two-dimensional image into a series of small images according to a set pixel size; Step S42: input each small image into the trained full convolutional network model in sequence to perform pixel-by-pixel prediction, so as to obtain high-resolution isotropic two-dimensional image data of each small image; Step S43, stitching the predicted high-resolution small images into a complete large image to obtain a high-resolution isotropic zy surface image; a loop processing unit, used for loop processing all the zy plane and zx plane low-resolution two-dimensional image data of the three-dimensional image data set to obtain all the zy plane and zx plane high-resolution two-dimensional image data; The three-dimensional high-resolution result output unit is used to re-slice and restore the high-resolution two-dimensional image data of the zy plane and zx plane, and output a three-dimensional image data set of brain tissue with a high-resolution single-cell resolution level.

8. The system according to claim 7, characterized in that The full convolutional network model is composed of convolutional layers, maximum pooling layers and upsampling layers, so that it can extract information features of images and predict high-resolution images. By adjusting the parameters of the full convolutional network structure, the network model with the optimal parameters in the experimental test is selected. The optimal parameters are designed based on the number of convolutions, maximum pooling and upsampling of the network while ensuring the accuracy and similarity of the predicted image and taking reducing the time of network training and prediction as the criterion.

9. The system according to claim 7, characterized in that The step of “inputting the two-dimensional data sets A and B into the fully convolutional network model for training” in the fully convolutional network model training unit is specifically: Input a low-resolution two-dimensional dataset B, first generate a dataset B' through image preprocessing and upsampling, so that the dataset B' has the same size as the dataset A, ensuring that the input and output images of the full convolutional network model have the same size, and at the same time correspond one-to-one with the gold standard dataset A, use the full convolutional network model to learn the transformation from low-resolution data B' to high-resolution data A, and after repeated training, select the network model weight result with the best predicted image.

10. The system according to claim 7, characterized in that The z-direction high-resolution image prediction unit splits a single zy surface low-resolution two-dimensional image into a series of small images; each small image is sequentially input into a trained full convolutional network model for pixel-by-pixel prediction to obtain high-resolution isotropic two-dimensional image data of each small image; and then the large image is split into small images in the order in which the large image is split, and the redundancy value of the small image boundaries during splicing is set, and the small images are re-spliced ​​and restored into a complete zy surface large image, thereby obtaining a high-resolution isotropic zy surface image.

Images