A few-shot microscopic image digital refocusing method based on reference guided learning

Abstract

This invention discloses a few-sample digital refocusing method for microscopic images based on reference-guided learning, relating to the fields of optical microscopy and image processing. Its technical features include: addressing the digital refocusing requirements of defocused imaging in optical microscopy systems, capturing a stack of z-axis images of a thin sample B via z-axis scanning; capturing a defocused image of a thin sample A using the same optical microscopy system; selecting the defocused image with the closest blur from the z-axis image stack of sample B based on a blur criterion; constructing a neural network model; inputting the defocused images of thin samples A and B into the network, constructing a loss function to drive network optimization; and obtaining the refocused image of thin sample A and a fixed model of the neural network after optimization. Defocused images of similar thin samples at the same defocus distance can be rapidly refocused using this fixed model, while defocused images at different defocus distances can achieve refocusing by repeating the aforementioned process and further optimizing. The advantages of this method are that it can complete network optimization with only a small number of samples, has low data acquisition and network optimization costs, can achieve image deconvolution without measuring or estimating the system's point spread function, has good refocusing effect and robust generalization performance, and can be applied to any image refocusing scenario in optical microscopy imaging systems.

Description

Technical Field

[0001] This invention belongs to the field of optical microscopy and image processing, and specifically relates to a digital refocusing method for defocused images based on an optical microscopy system. Background Technology

[0002] Optical microscopy imaging systems are widely used in biological tissue research and precision industrial measurements. Accurate and rapid high-resolution imaging has always been one of the development directions of microscopy systems. However, in actual imaging processes, due to factors such as sample movement, lens offset, and system control errors, it is impossible to maintain a consistently focused image of the target, resulting in defocused images at unknown distances. Utilizing computational imaging models and image processing techniques to convert defocused images into focused images through digital refocusing is of great significance for optical microscopy imaging observation.

[0003] Optical microscopy imaging systems can be approximated as linear space-invariant systems, therefore the imaging modulation effect of the system on the target can be represented by the optical point spread function. Based on this, the actual imaging process can be modeled as the convolution of the ideal in-focus image of the target with the system's point spread function. Thus, the refocusing process from the defocused image to the in-focus image is transformed into an image deconvolution process. For this incoherent light imaging process, traditional image restoration algorithms are widely used, such as the Wiener filter, the Lucy-Richardson algorithm, and the Land-Weber algorithm. These methods, combined with manually selected prior information such as total variation, sparsity, and nonnegativity, can further improve performance. However, in practical applications, they are often limited by factors such as accurate spread function estimation, experience-based parameter settings, and time-consuming and unstable iterative processes. In recent years, deep learning technology has developed rapidly and achieved results far exceeding those of traditional algorithms in image deconvolution. However, most of these neural networks are based on supervised learning, thus facing obstacles such as the acquisition of large amounts of training data and paired labels, lengthy optimization processes, and limited generalization ability. Some neural network methods based on self-supervised learning can get rid of the cost problem of large amounts of data and labels, but at the cost of accuracy and fast response in deconvolution.

[0004] Therefore, proposing an image deconvolution algorithm that meets the needs and requirements of practical applications to achieve digital refocusing is of great significance. This method needs to meet the following criteria: 1. Low data acquisition cost; 2. Fast optimization process; 3. No need to measure or estimate the system's point spread function; 4. Good deconvolution effect; 5. Applicability to different imaging systems, different samples, and different defocus distances. Summary of the Invention

[0005] This invention provides a few-sample digital refocusing method for microscopic images based on reference-guided learning. This method addresses the digital refocusing requirements of defocused imaging in optical microscopy systems. It acquires a stack of z-axis images of a thin sample B using z-axis scanning as prior information for the system's imaging, which participates in network training in a reference-guided manner. A defocused image of a thin sample A is then captured using the same optical microscopy system. Based on a blur criterion, the defocused image with the closest blur from the z-axis image stack of sample B is selected. A neural network model is constructed. The defocused images of thin samples A and B are input into the network, and distance loss constraints and regularization loss constraints are constructed to drive the optimization of network parameters. After optimization, a refocused image of thin sample A and a fixed model of the neural network are obtained. Defocused images of similar thin samples at the same defocus distance can be rapidly refocused using this fixed model, while defocused images at different defocus distances can achieve refocusing by repeating the aforementioned process and further optimization. The advantages of this method are that it only requires a small number of samples to complete the network optimization, the data acquisition cost and network optimization cost are low, image deconvolution can be achieved without measuring or estimating the system's point spread function, it has good refocusing effect and robust generalization performance, and it can be applied to any image refocusing scenario in optical microscopy imaging systems.

[0006] The specific technical solution of this invention is as follows:

[0007] A few-shot digital refocusing method for microscopic images based on reference-guided learning, characterized by the following steps:

[0008] S1. For a certain optical microscopic imaging system, a stack of z-axis images of a thin sample B is captured by z-axis scanning, which includes the in-focus image of the thin sample B and the defocus images at different distances;

[0009] S2. Using the same optical microscopic imaging system, replace thin sample B with a similar thin sample A, and take a single defocused image of thin sample A at an unknown defocus distance;

[0010] S3. Based on the blur criterion, select an out-of-focus image from the z-axis image stack of thin sample B that has the closest blur to the out-of-focus image of thin sample A, and select the in-focus image of thin sample B to pair with the out-of-focus image as a reference guide.

[0011] S4. Establish a neural network model for digital refocusing of defocused microscopic images;

[0012] S5. Using the out-of-focus images of thin samples A and B as input, the in-focus image of thin sample B as label, and the refocused images of thin samples A and B as output, construct loss function constraints to drive parameter optimization of the neural network model;

[0013] S6. After sufficient optimization, a refocused image of thin sample A and a fixed model of the neural network can be obtained.

[0014] The optical microscopy system in step S1 can be any incoherent optical microscopy system that satisfies the theory of linear optical systems, including but not limited to bright-field optical microscopy systems and wide-field fluorescence microscopy systems.

[0015] The z-axis scanning in step S1 can be performed electrically or manually, and the smaller the scanning interval, the better.

[0016] The thin sample in step S1 specifically refers to a sample whose thickness does not exceed the system's depth of field range, meaning that the system can capture a fully focused image of the thin sample.

[0017] The same optical microscopic imaging system in step S2 represents the thin samples A and B undergoing the same imaging system response.

[0018] The similar thin sample in step S2 specifically refers to the thin sample B that has a high degree of similarity in image features, including but not limited to the same biological cells, similar tissue slices, and samples with similar microstructures.

[0019] The unknown defocus distance in step S2 specifically refers to any defocus distance within the distance range of the z-axis image stack.

[0020] The blur criterion in step S3 can be any calculation method for quantifying the degree of image blur, including but not limited to Fourier ring correlation and image gradient.

[0021] The neural network model in step S4 can be any structured network model, including but not limited to U-shaped neural networks, fully convolutional networks, Transformer networks, and custom networks for model enhancement.

[0022] The beneficial effects of this invention lie in its proposal of a few-sample digital refocusing method for microscopic images based on reference-guided learning, addressing the digital refocusing requirements of optical microscopic imaging systems. Compared to traditional image restoration algorithms, this method eliminates the need to estimate the system's point spread function as the deconvolution kernel and achieves better image deconvolution results without relying on manual parameter tuning. Compared to supervised learning neural network methods, this method avoids the need for extensive data and label collection for training, thus mitigating the lengthy optimization process and significantly reducing the cost of using neural networks while maintaining equally excellent image deconvolution results. Compared to self-supervised learning neural network methods, this method achieves superior, more stable, and more robust digital refocusing results while maintaining a certain level of rapid response capability. This method simultaneously satisfies the characteristics of low data acquisition cost, fast optimization process, no need to measure or estimate the system's point spread function, good deconvolution results, and applicability to different imaging systems, different samples, and different defocus distances. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0024] Figure 1 This is a diagram illustrating the main steps of a few-sample digital refocusing method for microscopic images based on reference-guided learning in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the convolutional neural network structure based on the Lucy-Richardson algorithm in an embodiment of the present invention;

[0026] Figure 3 This is a flowchart illustrating the data acquisition, data processing, and network optimization processes in an embodiment of the present invention.

[0027] Figure 4 This diagram illustrates the comparison of bright-field defocused images of astrocytes and fluorescence defocused images of breast cancer cells at a defocus distance of 3μm, as provided in this embodiment of the invention, after being restored using Wiener filters, the Land-Weber algorithm, the Lucy-Richardson algorithm, a supervised neural network model, a self-supervised neural network model, and the algorithm involved in this invention.

[0028] Figure 2 In the diagram: I represents the input defocused image, R represents the output refocused image, and D, FP, and DV represent the intermediate feature maps of the network. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0030] A few-shot digital refocusing method for microscopic images based on reference-guided learning, characterized by the following steps:

[0031] S1. For a certain optical microscopic imaging system, a stack of z-axis images of a thin sample B is captured by z-axis scanning, which includes the in-focus image of the thin sample B and the defocus images at different distances;

[0032] S2. Using the same optical microscopic imaging system, replace thin sample B with a similar thin sample A, and take a single defocused image of thin sample A at an unknown defocus distance;

[0033] S3. Based on the blur criterion, select an out-of-focus image from the z-axis image stack of thin sample B that has the closest blur to the out-of-focus image of thin sample A, and select the in-focus image of thin sample B to pair with the out-of-focus image as a reference guide.

[0034] S4. Establish a neural network model for digital refocusing of defocused microscopic images;

[0035] S5. Using the out-of-focus images of thin samples A and B as input, the in-focus image of thin sample B as label, and the refocused images of thin samples A and B as output, construct loss function constraints to drive parameter optimization of the neural network model;

[0036] S6. After sufficient optimization, a refocused image of thin sample A and a fixed model of the neural network can be obtained.

[0037] The optical microscopy system in step S1 can be any incoherent optical microscopy system that satisfies the theory of linear optical systems, including but not limited to bright-field optical microscopy systems and wide-field fluorescence microscopy systems.

[0038] The z-axis scanning in step S1 can be performed electrically or manually, and the smaller the scanning interval, the better.

[0039] The thin sample in step S1 specifically refers to a sample whose thickness does not exceed the system's depth of field range, meaning that the system can capture a fully focused image of the thin sample.

[0040] The same optical microscopic imaging system in step S2 represents the thin samples A and B undergoing the same imaging system response.

[0041] The similar thin sample in step S2 specifically refers to the thin sample B that has a high degree of similarity in image features, including but not limited to the same biological cells, similar tissue slices, and samples with similar microstructures.

[0042] The unknown defocus distance in step S2 specifically refers to any defocus distance within the distance range of the z-axis image stack.

[0043] The blur criterion in step S3 can be any calculation method for quantifying the degree of image blur, including but not limited to Fourier ring correlation and image gradient.

[0044] The neural network model in step S4 can be any structured network model, including but not limited to U-shaped neural networks, fully convolutional networks, Transformer networks, and custom networks for model enhancement.

[0045] Example 1: The workflow of a few-shot digital refocusing method for microscopic images based on reference-guided learning is as follows:

[0046] according to Figure 1 As described in step S1, thin sample B is observed using a bright-field or fluorescence microscopy system under a 40x microscope objective with a numerical aperture of 0.6. Here, the bright-field microscopy system observes astrocyte slides, while the fluorescence microscopy system observes breast cancer cell slides. The system controls the thin sample B to achieve complete focus imaging using either manual observation or an autofocus algorithm. A z-axis image stack is acquired at 0.1μm intervals, starting from the in-focus position and ranging from -5μm to 5μm, using a z-axis scanning method. This defocus stack contains 100 defocused images at different distances and 1 in-focus image.

[0047] according to Figure 1 As described in step S2, thin sample B is replaced with a similar thin sample A. Here, it is replaced with another astrocyte slide and a breast cancer cell slide. A defocused image is captured at any distance within the range of -5 μm to 5 μm using the same bright-field or fluorescence microscopy imaging system. During this process, the illumination intensity of the light source and the system imaging parameters such as camera exposure time and ISO must remain constant.

[0048] according to Figure 1 As described in step S3, Fourier ring correlation is used as the blur criterion to perform blur quantification calculation on the z-axis image stack of thin sample B. The blur metric of the out-of-focus image of thin sample A is calculated, and the out-of-focus image with the closest blur metric is selected from the z-axis image stack of thin sample B. A focused image of thin sample B is then selected for pairing as a reference guide. At this point, the data required for neural network model optimization is ready.

[0049] according to Figure 1As described in step S4, a neural network model for digital refocusing of defocused microscopic images is established. (As...) Figure 2 As shown, this neural network model adopts the framework of the Lucy-Richardson algorithm, replacing the point spread function with multiple convolutional kernels. The iterative steps of the Lucy-Richardson algorithm are as follows:

[0050]

[0051] In the formula, i and i+1 are the iteration steps, and H is the point spread function of the imaging process. * The matrix H is a flip of I, which is the defocused image, and R is the matrix of H. i It is the refocused image obtained in the i-th iteration. This is a convolution operation, where D is the background term, which can be zero or other constants. Therefore, the overall structure of this neural network model can be simplified as follows:

[0052] (1)FP=Conv(I)+D, (2)DV=I / FP, (3)BP=Conv(DV), (4)R=I·BP

[0053] In the formula, D, FP, DV, and BP are intermediate feature maps generated corresponding to the Lucy-Richardson algorithm steps, Conv() is the convolution operation, I is the input defocused image, and R is the output refocused image. The neural network model uses convolution kernels of different sizes, 3×3 and 25×25, and their positions are as follows... Figure 2 As shown, after each convolution operation, an activation function called Softplus is applied to limit the numerical range.

[0054] according to Figure 1 As described in step S5, the out-of-focus image of thin sample A and the selected out-of-focus image of thin sample B are used as network inputs, and the in-focus image of thin sample B is used as the network label. The network response outputs the refocused images of thin samples A and B. A distance loss function constraint is constructed using the out-of-focus and in-focus images of thin sample B, while a regularization loss constraint is applied to the refocused images of thin samples A and B. The specific formula for calculating the loss function is as follows:

[0055] Distance loss = MSE(refocused image of thin sample B, in-focus image of thin sample B)

[0056] Regularized loss = ∑G(refocused image of thin sample A) + ∑G(refocused image of thin sample B)

[0057] In the formula, MSE() represents the mean squared error of two images, and G() represents the gradient of the image. Using a combination of distance loss constraints and regularization loss constraints, the neural network model's parameters are optimized through gradient backpropagation, with an iteration count of 2000 steps. The hardware and software used for optimization are: Windows 10 operating system, Intel(R) Core i7-10700K CPU, one Nvidia Geforce GTX 1080Ti, 32.0GB RAM, Python 3.7 language environment, and PyTorch 1.12.0 deep learning framework. The initial learning rate of the neural network model is set to 0.004, and it is set to decrease to 70% of its original value every 400 steps. The optimizer is Adam.

[0058] according to Figure 1 As described in step S6, after a certain period of training iterations, an optimized network solidification model and a refocused image of thin sample A can be obtained. In the same optical microscopy imaging system, using the solidification model of the neural network, a digital refocusing function can be achieved for similar thin samples at the same defocus distance. For similar thin samples at different defocus distances, steps S2 to S6 can be repeated to achieve the refocusing function through network re-optimization.

[0059] To further demonstrate the practical performance of the reference-guided learning-based digital refocusing method for few-sample microscopic images involved in this invention, 3μm defocused images of astrocytes and breast cancer cells were obtained under simulated bright-field and fluorescence microscopy imaging systems, respectively. Figure 4 The refocusing performance of Wiener filters, the Land-Weber algorithm, the Lucy-Richardson algorithm, supervised neural network models, self-supervised neural network models, and the algorithm of this invention were compared in two cases. Notably, the Wiener filter, the Land-Weber algorithm, and the Lucy-Richardson algorithm all use the system's point spread function as the deconvolution kernel. The supervised neural network model was trained on a dataset of 1000 pairs of defocused and in-focus images. Figure 4As can be seen, the refocused image obtained by the algorithm of this invention has a visual effect far superior to the defocused image and is basically consistent with the in-focus image. In the local magnification comparison of the refocused image, the algorithm of this invention has the best ability to preserve details and can completely suppress defocus blur. Peak signal-to-noise ratio and structural similarity are used as evaluation indicators to quantify the effect of refocusing, and the comparison results are shown in Table 1. After processing by the algorithm of this invention, the peak signal-to-noise ratio and structural similarity of astrocytes can be improved from 20.03 and 0.530 in the defocused image to 27.48 and 0.951 in the refocused image, and the peak signal-to-noise ratio and structural similarity of breast cancer cells can be improved from 30.83 and 0.901 in the defocused image to 36.34 and 0.983 in the refocused image. The processing time of the algorithm for the 512×512 pixel defocused image is also shown in Table 1. The algorithm involved in this invention can guarantee the refocusing effect while completing the optimization in only about 4 minutes. Compared with self-supervised and supervised neural network models, the data cost and time cost are significantly reduced.

[0060] In summary, this invention relates to a few-sample digital refocusing method for microscopic images based on reference-guided learning, which demonstrates excellent performance in refocusing defocused images in optical microscopy imaging systems. The algorithm involved in this invention requires only a small number of images to optimize the neural network model, resulting in low data acquisition costs and a fast optimization process. Stable and good deconvolution results are achieved without measuring or estimating the system's point spread function throughout the process, making it applicable to different imaging systems, different samples, and different defocus distances.

[0061] Table 1 Comparison of Quantitative Evaluation Indicators for Different Algorithms

[0062]

Claims

1. A method for digital refocusing of few-sample microscopic images based on reference-guided learning, characterized in that... The method includes the following steps: S1. For a certain optical microscopic imaging system, a stack of z-axis images of a thin sample B is captured by z-axis scanning, which includes in-focus images of the thin sample B and out-of-focus images at different distances; S2. Using the same optical microscopic imaging system, replace thin sample B with a similar thin sample A, and take a single defocused image of thin sample A at an unknown defocus distance; S3. Based on the blur criterion, select an out-of-focus image from the z-axis image stack of thin sample B. The selected out-of-focus image of thin sample B has the closest blur to the out-of-focus image of thin sample A. Pair the in-focus image of thin sample B with the selected out-of-focus image of thin sample B as a reference guide. S4. Establish a neural network model for digital refocusing of defocused microscopic images; S5. Using the out-of-focus images of thin samples A and B as input, the in-focus image of thin sample B as the label, and the refocused images of thin samples A and B as output, a loss function constraint is constructed to drive the parameter optimization of the neural network model. The loss function includes distance loss and regularization loss, and the calculation formula is as follows: In the formula, MSE() represents the mean square error of two images, and G() represents the gradient of the image. S6. After sufficient optimization, the refocused image of thin sample A and the solidified model of the neural network are obtained.

2. The method for digital refocusing of few-sample microscopic images based on reference-guided learning according to claim 1, characterized in that, The optical microscopy system in step S1 is either a bright-field optical microscopy system or a wide-field fluorescence microscopy system.

3. The method for digital refocusing of few-sample microscopic images based on reference-guided learning according to claim 1, characterized in that, The z-axis scanning in step S1 can be performed electrically or manually.

4. The method for digital refocusing of few-sample microscopic images based on reference-guided learning according to claim 1, characterized in that, The thin sample in step S1 is a sample whose thickness does not exceed the system depth of field range.

5. A method for digital refocusing of few-sample microscopic images based on reference-guided learning according to claim 1, characterized in that, The similar thin sample in step S2 belongs to the same biological cells or tissue sections.

6. A method for digital refocusing of few-sample microscopic images based on reference-guided learning according to claim 1, characterized in that, The unknown defocus distance in step S2 is any defocus distance within the distance range of the z-axis image stack.

7. A method for digital refocusing of few-sample microscopic images based on reference-guided learning according to claim 1, characterized in that, The blur criterion in step S3 is used to quantify the degree of image blur, and the criterion is Fourier ring correlation or image gradient.

8. A method for digital refocusing of few-sample microscopic images based on reference-guided learning according to claim 1, characterized in that, The neural network model in step S4 is one of a U-shaped neural network, a fully convolutional network, or a Transformer network.

Images