Microscopic imaging virtual refocusing method based on sequence learning
Through the U-Net deep learning network autofocus, the problems of focus drift and manual focus in traditional microscope imaging are solved, efficient digital refocusing effect is achieved, and image quality is improved.
Patent Information
- Application Number
- CN202410274133.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-11
- Publication Date
- 2025-09-12
AI Technical Summary
Traditional microscope imaging requires manual focusing, which leads to focus drift, is time-consuming and labor-intensive, and makes it difficult to quickly form high-resolution images.
A U-net based deep learning network is used to achieve autofocus by training the difference between defocused and focused images, and digital refocusing is performed using the deep learning network.
Without changing the microscope hardware, image focusing is achieved automatically, which improves the SSIM and PSNR values of the image and reduces operation complexity and time cost.
Smart Images
Figure CN120635155A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computer vision technology in the information field, and in particular to a method for performing digital refocusing using deep learning in microscopic imaging. Background Art
[0002] For digital image processing tasks, deep learning methods generally use convolutional neural networks to extract image features. After forward propagation to obtain the target image or classification results, the loss function is calculated and the network parameters are updated through backpropagation. By reducing the loss function, the network is trained to achieve the purpose of digital image processing.
[0003] Digital refocusing research aims to address the limitations of traditional imaging systems, such as depth blur, occlusion, and resolution. Traditional imaging systems only capture the intensity of light reflected from an object's surface, while ignoring the other information carried by the light as it propagates. This results in a loss of image quality and information content, as well as inaccurate estimates of an object's depth and shape.
[0004] The most widely used method currently is to use light field imaging to achieve digital refocusing.
[0005] The imaging focus process of traditional microscopes requires manual focus. In the biological field, long-term use of the microscope will cause focus drift. When scanning and stitching multiple fields of view (FoV) to form a high-content, high-resolution image, a sample that is not flat needs to be refocused at each position. These operations require professional training and are time-consuming and labor-intensive.
[0006] The present invention can digitally refocus images with severe positive and negative defocus without changing the existing system of the microscope, and can obtain focused images using only a neural network. The refocused images have higher values than the real focused images in terms of SSIM and PSNR.
[0007] Furthermore, the present invention has no special requirements on hardware environment conditions and only requires the acquisition and processing of part of the data set in advance. Summary of the Invention
[0008] The present invention aims to overcome the focus drift caused by long-term use of microscopes in traditional imaging in the aforementioned prior art, the need for frequent manual focusing and multiple focusing required for scanning and stitching multiple fields of view, and provide a new intelligent algorithm for achieving autofocus using deep learning. A class of acquired focused and defocused images are paired, where the defocused image is used as the input image and input into the classic u-net deep learning network, and then the corresponding focused image is used as the label. Through continuous iterative training, the network learns the difference between the input image and the label image, which is then saved as the network weight, so that it can be implemented in subsequent tests without the need for training to achieve the effect of autofocus.
[0009] The specific technical solution of the present invention is:
[0010] A digital refocusing method for a microscope, characterized in that the method comprises the following steps:
[0011] S1. Data collection and preprocessing: cropping dataset images to 286*286 pixels to facilitate network training and reduce resource usage.
[0012] S2. Establishing a neural network model. This paper adopts the classic u-net model, which consists of two parts, downsampling and upsampling, to form a U-shaped network. This can reduce training time and GPU resource utilization, and obtain more accurate refocused images.
[0013] S3. Use cell scatter plots and tissue maps at different defocus distances, as well as label maps of corresponding focus positions for training, with a learning rate set to 1*10 -2 , weight decay is set to linear decay, the decay rate is set to 0.001, and the optimizer is Adam, where betas = (0.9, 0.999), eps = 1e-8;
[0014] S4. First, use the defocused and focused images in the dataset as input and label, respectively, and input them into the network for training. Then, use images with different defocus levels for testing. Calculate the structural similarity and signal-to-noise ratio of the output images, respectively. Visualize the train_loss and test_loss of the training and testing steps, respectively. The above results can be used to quantitatively judge the refocusing ability of the network.
[0015] The data set in step S1 can be cut into a smaller size, thereby saving training resources and reducing costs.
[0016] Other methods of step S1 may be reshooting or selecting from existing image data, which is characterized by low implementation cost.
[0017] The focused image and the defocused image in step S1 must be registered, and their structures and sharpness must correspond.
[0018] The network in step S2 can be other network models based on CNN structure, including but not limited to U-type neural network, GAN network, etc.
[0019] The types and parameters of the loss function and optimizer in step S3 can vary with different datasets and are not limited to MSELoss, Adam, etc.
[0020] The trained and saved network in step S4 needs to be tested with the results obtained by the network. The test is performed using images with different defocus levels. The structural similarity and signal-to-noise ratio of the output images are calculated respectively. The train_loss and test_loss of the training and test steps are visualized respectively. The above results can be used to quantitatively judge the refocusing ability of the network.
[0021] The beneficial effect of the present invention is that the imaging focusing process of traditional microscopes does not require manual focusing. For the biological field, long-term use of the microscope will cause focus drift. When scanning and splicing multiple fields of view (FoV) to form a high-content, high-resolution image, a sample that is not flat enough needs to be refocused at each position. These all require the operation of professionally trained personnel and are time-consuming and labor-intensive. The present invention can digitally refocus images with severe positive and negative defocus without changing the existing system of the microscope, and a focused image can be obtained by using only a neural network, and the refocused image has higher values in SSIM and PSNR than the real focused image. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:
[0023] Figure 1 : A flow chart of a refocusing method provided by an embodiment of the present invention;
[0024] Figure 2 : A focus-defocus comparison diagram during the network training phase provided by an embodiment of the present invention:
[0025] Figure 3 : This is the focus and defocus map output by the network during the test phase provided by the embodiment of the present invention:
[0026] Figure 4 : Loss diagram of the test phase after the training model provided by the embodiment of the present invention:
[0027] Figure 2Middle: The above input-image is the defocused image when the defocus distance is maximum (z=1).
[0028] Figure 3 In the figure, z=number represents the defocused image input at different defocus distances. DETAILED DESCRIPTION
[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.
[0030] A digital refocusing method for a microscope, characterized in that the method comprises the following steps:
[0031] S1. Data collection and preprocessing: cropping dataset images to 286*286 size images to facilitate network training and reduce resource usage;
[0032] S2. Establishing a neural network model. This paper adopts the classic u-net model, which consists of two parts, downsampling and upsampling, to form a U-shaped network. This can reduce training time and GPU resource utilization, and obtain more accurate refocused images.
[0033] S3. Use cell scatter plots and tissue maps at different defocus distances, as well as label maps of corresponding focus positions for training, with a learning rate set to 1*10 -2 , weight decay is set to linear decay, the decay rate is set to 0.001, and the optimizer is Adam, where betas = (0.9, 0.999), eps = 1e-8;
[0034] S4. First, use the defocused and focused images in the dataset as input and label, respectively, and input them into the network for training. Then, use images with different defocus levels for testing. Calculate the structural similarity and signal-to-noise ratio of the output images, respectively. Visualize the train_loss and test_loss of the training and testing steps, respectively. The above results can be used to quantitatively judge the refocusing ability of the network.
[0035] The data set in step S1 can be cut into a smaller size, thereby saving training resources and reducing costs.
[0036] Other methods of step S1 may be reshooting or selecting from existing image data, which is characterized by low implementation cost.
[0037] The focused image and the defocused image in step S1 must be registered, and their structures and sharpness must correspond.
[0038] The network in step S2 can be other network models based on CNN structure, including but not limited to U-type neural network, GAN network, etc.
[0039] The types and parameters of the loss function and optimizer in step S3 may vary with different data sets and are not limited to MSELoss, Adam, etc.
[0040] The trained and saved network in step S4 needs to be tested with the results obtained by the network. The test is performed using images with different defocus levels. The structural similarity and signal-to-noise ratio of the output images are calculated respectively. The train_loss and test_loss of the training and test steps are visualized respectively. The above results can be used to quantitatively judge the refocusing ability of the network.
[0041] Example 1: The workflow of a method for digital refocusing applied to a microscope is as follows:
[0042] As described in step S1, this experiment directly feeds our pre-trained model with out-of-focus images from the BBBC006 dataset. Each image is resized to the same size of 572*572 or 286*286, and images with low sample rates and high noise are removed to improve dataset quality and reduce computing resource usage and cost.
[0043] According to step S2, a neural network model is established. The network includes two parts: downsampling to extract features and upsampling to restore images. It is a typical U-shaped network structure. Figure 1 As shown in the figure, the networks all use a deep convolutional neural network with a mixed 1×1 and 3×3 convolution kernel size and a U-shaped network structure. In addition, pixel-level addition operations, image channel-scale concatenation operations, and the activation function LeakyReLU and batch normalization are also used to assist in network construction.
[0044] According to step S3, the network parameters are adjusted first, that is, the loss function of the network, the selection of the optimizer and the adjustment of the parameters are adjusted according to the different input data sets and different tasks. The input defocused image is subjected to multiple Figure 1 In the convolution and pooling operations of the downsampling part shown, the size is getting smaller and the features are gradually extracted. After a certain number of downsampling layers, the image size and features are gradually restored through deconvolution or linear interpolation. Finally, after a full connection or convolution, the original size of the image can be restored.
[0045] The specific calculation formula of the network loss function is:
[0046] Loss = MSE (trained input image, label image)
[0047] Where MSE(·) represents the mean square error of two images.
[0048]
[0049] p i =x i w+b
[0050] After network optimization is completed as described in step S4, network training is then performed as described in step S3. The network compares the difference between the network output image and the labeled image. This difference is calculated using a loss function, and the calculated difference is backpropagated to the network for the next training round. This gradually allows the network to learn the mapping relationship between the input image and the labeled image, thereby completing network training for 650 epochs. During the iterative process, the order of the defocused image and the labeled image is randomly shuffled before each iteration.
[0051] The hardware and software used in the method described in this invention are as follows: The training software and hardware equipment used were: four GeForce 2080Ti graphics cards, Ubuntu 18.04.3 operating system, Python 3.7 programming language, PyTorch 1.4.0 deep learning framework, and PyCharm compilation environment. The network's initial learning rate was 0.001, and the learning rate was set to decrease to 95% of its original value every 100 steps. The optimizer used was Adam.
[0052] In summary, the present invention relates to a digital refocusing method applied to a microscope, which aims to overcome the defects of the aforementioned prior art such as focus drift caused by long-term use of the microscope in traditional imaging, frequent manual focusing, and multiple focusing steps required for scanning and stitching multiple fields of view. A new intelligent algorithm for achieving autofocus using deep learning is provided, which can digitally refocus images with severe positive and negative defocus without changing the existing system of the microscope, and can obtain a focused image using only a neural network. The refocused image has higher values than the true focused image in terms of SSIM and PSNR.
[0053] In actual application scenarios, a large amount of supervised learning neural network algorithms with labeled data are required. The method involved in the present invention has a good digital refocusing effect. After testing, both SSIM and PSNR have high values, and the cost of use is low. It is suitable for imaging of different systems and different observation targets.
Claims
1. A digital refocusing method for a microscope, characterized in that The method comprises the following steps: S1. This experiment uses the out-of-focus images from the BBBC006 dataset to directly input our pre-trained model. The out-of-focus image dataset of the BBBC006 dataset is divided into a training set and a test set for subsequent training and testing. S2. Resize each image to the same size of 572*572 or 286*286, and delete images with low sample ratio and high noise to improve the quality of the dataset; S3. Build an object detection network based on the U-net model, set the image size to 286x286, and use the Adam optimization algorithm. Train the network using out-of-focus images from the BBBC006 dataset and tune the hyperparameters. Input the network into a test set of images from the dataset to generate a refocused image. Use the SSIM and PSNR metrics to measure the difference between the refocused image and the original focused image. S4. Train the network using the experimental training set and save the trained weight parameters. Use the trained model to test other out-of-focus images in the dataset and save the test results and test loss curve. S5. The hardware and software used for training are: four GeForce 2080Ti graphics cards, Ubuntu 18.04.3 operating system, Python 3.7 programming language, PyTorch 1.4.0 deep learning framework, and PyCharm compilation environment.
2. The digital refocusing method for a microscope according to claim 1, characterized in that: The data set in step S1 can realize virtual refocusing on the trained network.
3. The digital refocusing method for a microscope according to claim 1, characterized in that: The data set of step S2 can be resized to reduce the usage of training resources and thus achieve faster refocusing.
4. The digital refocusing method for a microscope according to claim 1, characterized in that: The network model used in step S3 requires a set of paired defocused and focused images as input and labels, respectively.
5. The digital refocusing method for a microscope according to claim 1, characterized in that: The network in step S3 may be a network model of any structure, including but not limited to a U-shaped neural network, a fully connected network, a fully convolutional network, a Transformer network, etc.
6. The digital refocusing method for a microscope according to claim 1, characterized in that: The trained network model of steps S4 and S5 can be directly applied to microscopic images without the need for additional training and manual operation to restore defocused images. The network can be trained without requiring very high computing power.