Method for expanding depth of field of optical operation microscope

Abstract

The invention relates to a depth-of-field expansion method for an optical surgical microscope. The method comprises the following steps: collecting an open-source high-definition high-frequency image database; acquiring point spread functions corresponding to different focus depths of the operating microscope; constructing a random depth surface type database; simulating a partial out-of-focus image of the non-flat sample imaged by the optical microscope; simulating a perfect focusing image of the flat sample imaged by the optical microscope; combining a large number of paired data partial out-of-focus images-perfect focusing images; and constructing an encoder-decoder structure network model from the image end to the image end to carry out model training and application. The problem that high-quality pairing training data conforming to imaging of the operating microscope is difficult to obtain for model training is solved, partial defocusing / focusing simulation data of the operating microscope conforming to a real physical rule generation effect can be synthesized in batches, a high-quality training data set is constructed at low cost, the number, diversity and randomness of the training data are ensured, and the training efficiency is improved. And defocus information is efficiently learned and recovered while focusing information is kept.

Description

Technical Field

[0001] The invention relates to a biomedical imaging and computer vision technology, and in particular to a method for extending the depth of field of an optical surgical microscope. Background Art

[0002] An optical microscope is a precision instrument that uses optical principles to perform imaging. It uses the magnification imaging principle of a convex lens to achieve magnified observation of tiny objects through a combination of an objective lens and an eyepiece. Researchers and doctors use optical microscopes extensively to observe and quantify cell functions, organ development, or disease mechanisms. The optical resolution (Resolution) and imaging depth of field (Depth-Of-Field, DOF) of a microscope are two important parameters to consider during use. The relationship between the two is:

[0003]

[0004] Where, λ is the central wavelength of the incident light;

[0005] It can be seen that the higher the resolution, the shallower the imaging depth of field, which is largely hindered in clinical applications. For example, in a microscope equipped with a standard objective, achieving subcellular lateral resolution (about 2-3 μm) will limit the depth of field to about 30 μm. In order to achieve high-resolution histopathological imaging in medical testing applications, biopsy and surgical specimens are usually fixed and embedded in formalin, then cut into about 5 μm thick (to ensure that it is within the imaging depth of field) with a microtome, and then stained with medical dyes for evaluation under an optical microscope.

[0006] Optical surgical microscopes play a vital role in performing microsurgery, and the challenge of depth of field is even more significant in intraoperative microscopy. Figure 1 As shown in the figure, during intraoperative microscopy imaging, due to the non-flat surface of the tissue to be observed, the non-flat area will cause the area in the image to be unclear, resulting in partial optical defocusing of the field of view, that is, due to the change in the distance between the objective lens and the sample or inaccurate focusing, the image will be blurred or unclear, making positioning and visual interpretation difficult. Therefore, the extended depth of field of the microscope is needed to make it compatible with a deeper range of imaging depth.

[0007] The formation of microscopic images can be modeled by optical diffraction, where sharp point-like objects are imaged as blurred points after diffraction. Due to the existence of optical diffraction, it cannot focus light to a perfect point, but spreads the light into a diffraction pattern described by the point spread function (PSF). Since the image is formed by superimposing the contributions of all points in the object, a clearer image can be estimated by using the knowledge of the local diffraction pattern that summarizes the optical system and its aberrations.

[0008] In response to the contradiction between the depth of field and high resolution requirements of optical surgical microscopes, some methods have been developed to achieve microscope depth of field extension or defocus reconstruction applications. Kim et al. used blind deconvolution to directly restore the defocused blurred image with high resolution, but due to the lack of prior knowledge of the system point spread function, its performance is usually affected. In addition, the deconvolution processing speed cannot match the high-throughput data stream acquisition of the microscope (Scientific reports, 2015, 5(1), 9894). With the rapid development of artificial intelligence technology, data-driven methods are significantly superior to traditional methods such as deconvolution in terms of correction accuracy and processing speed. Shajkofci et al. used a convolutional neural network to estimate the spatially varying point spread function, and then used a deconvolution algorithm based on the estimated fixed spread function to deblur the defocused image to achieve defocus reconstruction. Compared with blind deconvolution, the accuracy is improved, but because the estimated point spread function is approximately an ideal Gaussian model and there is an error between the estimated value and the true value, the results need to be further improved (IEEE Transactions on Image Processing, 2020, 29, 5848-5861). Furthermore, Jin et al. first collected the three-dimensional point spread function of the microscope, applied it to each position on the focused image as the gold standard to obtain the overall defocused image, and then used the U-Net network widely used in deep learning to perform mapping training from the defocused blurred image to the focused image to achieve depth of field extension. However, since the defocus degree of the generated defocused image is overall consistent, it is not random and diverse, and is not suitable for simulating different degrees of defocus caused by tissue ups and downs in the same field of view of a surgical microscope. In addition, a large number of focus data of target samples need to be collected in advance as the gold standard for training, which leads to heavy data acquisition burden and poor generalization performance (PNAS, 2020, 117(52), 33051-33060).

[0009] The current lack of research on methods for extending the depth of field of optical surgical microscopes is reflected in the difficulty in obtaining high-quality paired training data that is consistent with surgical microscope imaging for model training: on the one hand, the actual paired defocused-in-focus microscopic data is time-consuming and laborious, and it is difficult to accommodate the diversity and randomness of training samples, resulting in insufficient model generalization capabilities; on the other hand, the training data generated by synthetic data is difficult to conform to the physical imaging laws, and it is difficult to meet the random defocusing effect of imaging samples close to that of real surgical microscopes. Considering the cost and quality of obtaining training data, it is difficult to guarantee the defocus recovery effect of the model on microscopic images, so the depth of focus extension capability of surgical microscopes needs to be further improved. Summary of the invention

[0010] Aiming at the problem that it is difficult to obtain high-quality paired training data that meets the requirements of surgical microscope imaging for model training, a method for extending the depth of field of optical surgical microscope is proposed.

[0011] The technical solution of the present invention is:

[0012] A method for extending the depth of field of an optical surgical microscope includes three stages: physics-based guided training data synthesis, model training, and model application, specifically the following steps:

[0013] S1: Collect open source high-definition and high-frequency image database;

[0014] S2: Collect the point spread function corresponding to different focal depths of the surgical microscope;

[0015] S3: Build a random depth face database; scale the face depth range to the sampling depth range of the acquired point spread function;

[0016] S4: simulates the partially out-of-focus image of an optical microscope imaging a non-flat sample; randomly selects an image from S1 as the structure to be imaged; randomly selects a surface type from S3 as the surface depth information of the structure to be imaged, and scales the surface type size to be consistent with the data size of the structure to be imaged; according to the corresponding depth of each position of the structure to be imaged, a convolution operation is performed with the point spread function of the corresponding depth in S2 to simulate the microscopic imaging effect of the corresponding position after diffraction of the microscope light path; after convolution operation of all positions, a partially out-of-focus image corresponding to this surface type and the structure to be imaged is obtained, and after adding Poisson distribution and Gaussian distribution noise to the partially out-of-focus image, the final simulated partially out-of-focus image is obtained;

[0017] S5: Simulate the perfectly focused image of an optical microscope imaging a flat sample; the same image selected in S4 is selected from S1 as the structure to be imaged, and the surface depth corresponding to this structure is regarded as completely flat, so each position of the structure to be imaged is convolved with the point spread function corresponding to the 0 depth in S2, i.e., the focal plane position, to simulate the microscopic imaging effect of the corresponding position after diffraction. After the convolution operation is performed on all positions, a completely flat surface and a fully focused image corresponding to the structure to be imaged are obtained, and after adding Poisson distribution and Gaussian distribution noise to the fully focused image, the final simulated fully focused image is obtained;

[0018] S6: combining the partially out-of-focus image and the perfectly focused image generated by steps S4 and S5 of the same structure to be imaged as paired data required for model training, and by repeating steps S4 and S5, generating a large amount of paired data for subsequent model training;

[0019] S7: Model training: Build an image-to-image encoder-decoder structure network model, iteratively optimize based on the gradient descent method, and train the model to make the output result close to a fully focused image, that is, it has the ability to recover from defocus;

[0020] S8: Model application: The intraoperative microscopic imaging results of the surgical microscope are input into the trained model. The model output is the focus image after defocus recovery, which realizes the depth of field extension of the surgical microscope.

[0021] Furthermore, the specific implementation steps are as follows:

[0022] S1: Collect an open source high-definition high-frequency image database, including images with rich texture or structural details and high-frequency information;

[0023] S2: Collect the point spread function corresponding to different focal depths of the surgical microscope; specifically:

[0024] An opaque microsphere with a diameter smaller than the resolution of the surgical microscope is used to approximate an ideal point light source image. The focal point of the objective lens is at the center of the microsphere as the focal plane position 0, and the original depth of field is set to ±aμm. The objective lens focus is moved up and down ±bμm to sample the corresponding images at different positions, where b>a. After image cropping, background removal, pixel value inversion, and pixel normalization, the corresponding Airy disk image is obtained, that is, the three-dimensional point spread function of the microscope optical system.

[0025] S3: Build a random depth face database; scale the face depth range to the sampling depth range of the acquired point spread function;

[0026] Constructing a random depth face database, specifically: constructing a random two-dimensional frequency domain distribution, dividing the amplitude of each frequency component by the total amplitude to obtain the relative amplitude of the frequency component, calculating the power spectrum density of each frequency component, thereby obtaining a two-dimensional power spectrum density, performing a discrete Fourier transform on the two-dimensional power spectrum density, obtaining a simulated artificial random face with random depth and surface roughness, and repeating this process to construct a random depth face database;

[0027] S4: simulates the partially out-of-focus image of an optical microscope imaging a non-flat sample; randomly selects an image from S1 as the structure to be imaged; randomly selects a surface type from S3 as the surface depth information of the structure to be imaged, and scales the surface type size to be consistent with the data size of the structure to be imaged; according to the corresponding depth of each position of the structure to be imaged, a convolution operation is performed with the point spread function of the corresponding depth in S2 to simulate the microscopic imaging effect of the corresponding position after diffraction of the microscope light path; after convolution operation of all positions, a partially out-of-focus image corresponding to this surface type and the structure to be imaged is obtained, and after adding Poisson distribution and Gaussian distribution noise to the partially out-of-focus image, the final simulated partially out-of-focus image is obtained;

[0028] S5: Simulate the perfectly focused image of an optical microscope imaging a flat sample; the same image selected in S4 is selected from S1 as the structure to be imaged, and the surface depth corresponding to this structure is regarded as completely flat, so each position of the structure to be imaged is convolved with the point spread function corresponding to the 0 depth in S2, i.e., the focal plane position, to simulate the microscopic imaging effect of the corresponding position after diffraction. After the convolution operation is performed on all positions, a completely flat surface and a fully focused image corresponding to the structure to be imaged are obtained, and after adding Poisson distribution and Gaussian distribution noise to the fully focused image, the final simulated fully focused image is obtained;

[0029] S6: combining the partially out-of-focus image and the perfectly focused image generated by steps S4 and S5 of the same structure to be imaged as paired data required for model training, and by repeating steps S4 and S5, generating a large amount of paired data for subsequent model training;

[0030] S7: Model training;

[0031] The model training is as follows: construct an encoder-decoder structure network model from image to image; use the partially out-of-focus images of the paired data as model input after image preprocessing, use the fully focused images as labels and the model output results to calculate the loss function, and iterate based on the gradient descent method. After training with massive synthetic paired data, the model output results are close to fully focused images, that is, they have the ability to recover from defocus;

[0032] S8: Model application;

[0033] The specific application of the model is: input the intraoperative microscopic imaging results of the surgical microscope into the trained model, and the model output result is the focus image after defocus recovery, which realizes the depth of field extension of the surgical microscope.

[0034] Preferably, in step S1, the image may be a natural texture image or a high-definition microscopic image.

[0035] Preferably, in step S7, the encoder-decoder structure network model can be U-Net or ResU-Net.

[0036] The beneficial effects of the present invention are:

[0037] A batch synthesis method for partial defocus / in-focus simulation data of surgical microscopes is proposed, which can generate effects that conform to real physical laws. It is possible to build a high-quality training data set at low cost, and ensure the quantity, diversity and randomness of the training data. It can efficiently learn to restore the defocus information while maintaining the focus information, and achieve low-cost, high-accuracy, and real-time defocus recovery effect of surgical microscopes, which expands the imaging depth of field and plays a key role in improving surgical precision and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram showing the imaging limitation of intraoperative tissues due to the limited depth of field of the optical surgical microscope of the present invention;

[0039] Figure 2 It is a schematic diagram of the process of the method for extending the depth of field of an optical surgical microscope of the present invention;

[0040] Figure 3 The present invention collects three views of the point spread function corresponding to different focal depths of the surgical microscope;

[0041] Figure 4 A surface display diagram of random depth and roughness generated by the present invention;

[0042] Figure 5 This is a diagram showing the defocus recovery effect of surgical microscopic images of the present invention. DETAILED DESCRIPTION

[0043] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0044] A method for extending the depth of field of an optical surgical microscope includes three stages: physics-based guided training data synthesis, model training, and model application. Figure 2 As shown;

[0045] Among them, the synthesis of training data based on physical guidance specifically includes:

[0046] S1: Collect open source high-definition and high-frequency image databases, mainly referring to images with rich texture or structural details and high-frequency information, such as natural texture images, high-definition microscopic images, etc.

[0047] S2: Collect the point spread function corresponding to different focal depths of the surgical microscope. Specifically:

[0048] An opaque microsphere with a diameter smaller than the resolution of the surgical microscope was used to approximate an ideal point light source image. The objective lens focus was set at the center of the microsphere as the focal plane position 0, the original depth of field was set to ±aμm, and the objective lens focus was moved up and down by ±bμm (b>a) to sample the corresponding images at different positions. After image cropping, background removal, pixel value inversion, and pixel normalization, the corresponding Airy disk image was obtained, that is, the three-dimensional (different focal depths) point spread function of the microscope optical system.

[0049] S3: Build a random depth face database and scale the face depth range to the sampling depth range of the collected point spread function.

[0050] Construct a random depth face database, specifically: construct a random two-dimensional frequency domain distribution, divide the amplitude of each frequency component by the total amplitude to obtain the relative amplitude of the frequency component, calculate the power spectral density of each frequency component, thereby obtaining the two-dimensional power spectral density, perform discrete Fourier transform on the two-dimensional power spectral density, and obtain a simulated artificial random face with random depth and surface roughness. Repeat this process to construct a random depth face database.

[0051] S4: simulates the partially out-of-focus image of an optical microscope imaging a non-flat sample. Randomly select an image from S1 as the structure to be imaged; randomly select a surface from S3 as the surface depth information of the structure to be imaged, and scale the surface size to be consistent with the data size of the structure to be imaged; according to the corresponding depth of each position of the structure to be imaged, perform a convolution operation with the point spread function of the corresponding depth in S2 to simulate the microscopic imaging effect of the corresponding position after diffraction of the microscope light path; after performing the convolution operation on all positions, obtain the partially out-of-focus image corresponding to this surface and the structure to be imaged, and add Poisson distribution and Gaussian distribution noise to the partially out-of-focus image to obtain the final simulated partially out-of-focus image.

[0052] S5: Simulate the perfect focus image of an optical microscope imaging a flat sample. The same image selected in S4 is selected from S1 as the structure to be imaged, and the surface depth corresponding to this structure is considered to be completely flat. Therefore, each position of the structure to be imaged is convolved with the point spread function corresponding to the 0 depth in S2, i.e., the focal plane position, to simulate the microscopic imaging effect of the corresponding position after diffraction. After convolution operation of all positions, a completely flat surface and a fully focused image corresponding to the structure to be imaged are obtained. After adding Poisson distribution and Gaussian distribution noise to the fully focused image, the final simulated fully focused image is obtained.

[0053] S6: The partially out-of-focus image and the perfectly focused image of the same structure to be imaged generated by steps S4 and S5 are combined as the paired data required for model training. By repeating steps S4 and S5, a large amount of paired data is generated for subsequent model training.

[0054] S7: Model training.

[0055] The specific model training is as follows: construct an image-to-image encoder-decoder structure network model, such as U-Net, ResU-Net, etc.; the partially out-of-focus images of the paired data are used as model input after image preprocessing, and the fully focused images are used as labels to calculate the loss function with the model output results, and iterative optimization is performed based on the gradient descent method. After training with massive synthetic paired data, the model output results are made close to the fully focused images, that is, they have the ability to recover from defocus.

[0056] S8: Model application.

[0057] The specific application of the model is: input the intraoperative microscopic imaging results of the surgical microscope into the trained model, and the model output result is the focus image after defocus recovery, which realizes the depth of field extension of the surgical microscope.

[0058] Experimental results:

[0059] The point spread function corresponding to different focal depths of the surgical microscope is collected. Figure 3 As shown, the middle position is the focus plane, and the three-dimensional point spread function of the surgical microscope objective imaging is collected along the circumferential direction with the focus plane as the center.

[0060] A random depth face database of 20,000 was constructed, and three typical face visualizations are shown below: Figure 4 As shown, it can be seen that the surface shapes can include different roughness levels and different depth shapes, which is diverse.

[0061] A database of 20,000 open-source high-definition and high-frequency images was collected to construct a diverse set of structures to be imaged.

[0062] The proposed training data synthesis method generated 20,000 pairs of partially out-of-focus images and fully in-focus images, thereby constructing a training dataset at low cost while ensuring the quantity and diversity of training data for model training.

[0063] The real microscopic images of surgical microscopes were input into the fully trained model for defocus recovery application testing. The microscopic image input and output results are shown in Figure 5 As shown, the first row is a real microscopic image, in which part of the image is out of focus due to different depths of the imaging surface, and the second row is the defocus recovery result. It can be seen that after model correction, the blurred structure part originally caused by defocus has improved clarity, while the structure part originally in focus has maintained its original morphology. The final result is approximately a fully focused microscopic image, which increases the focus range beyond the original small depth of field and expands the depth of field of the microscope.

[0064] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A method for extending the depth of field of an optical surgical microscope, characterized in that: It includes three stages: physics-based guided training data synthesis, model training, and model application. The specific steps are as follows: S1: Collect open source high-definition and high-frequency image database; S2: Collect the point spread function corresponding to different focal depths of the surgical microscope; S3: Build a random depth face database; scale the face depth range to the sampling depth range of the acquired point spread function; S4: simulates the partially out-of-focus image of an optical microscope imaging a non-flat sample; randomly selects an image from S1 as the structure to be imaged; randomly selects a surface type from S3 as the surface depth information of the structure to be imaged, and scales the surface type size to be consistent with the data size of the structure to be imaged; according to the corresponding depth of each position of the structure to be imaged, a convolution operation is performed with the point spread function of the corresponding depth in S2 to simulate the microscopic imaging effect of the corresponding position after diffraction of the microscope light path; after convolution operation of all positions, a partially out-of-focus image corresponding to this surface type and the structure to be imaged is obtained, and after adding Poisson distribution and Gaussian distribution noise to the partially out-of-focus image, the final simulated partially out-of-focus image is obtained; S5: Simulate the perfectly focused image of an optical microscope imaging a flat sample; the same image selected in S4 is selected from S1 as the structure to be imaged, and the surface depth corresponding to this structure is regarded as completely flat, so each position of the structure to be imaged is convolved with the point spread function corresponding to the 0 depth in S2, i.e., the focal plane position, to simulate the microscopic imaging effect of the corresponding position after diffraction. After the convolution operation is performed on all positions, a completely flat surface and a fully focused image corresponding to the structure to be imaged are obtained, and after adding Poisson distribution and Gaussian distribution noise to the fully focused image, the final simulated fully focused image is obtained; S6: combining the partially out-of-focus image and the perfectly focused image generated by steps S4 and S5 of the same structure to be imaged as paired data required for model training, and by repeating steps S4 and S5, generating a large amount of paired data for subsequent model training; S7: Model training: Build an image-to-image encoder-decoder structure network model, iteratively optimize based on the gradient descent method, and train the model to make the output result close to a fully focused image, that is, it has the ability to recover from defocus; S8: Model application: The intraoperative microscopic imaging results of the surgical microscope are input into the trained model. The model output is the focus image after defocus recovery, which realizes the depth of field extension of the surgical microscope.

2. The method for extending the depth of field of an optical surgical microscope according to claim 1, characterized in that: The specific implementation steps are as follows: S1: Collect an open source high-definition high-frequency image database, including images with rich texture or structural details and high-frequency information; S2: Collect the point spread function corresponding to different focal depths of the surgical microscope; specifically: An opaque microsphere with a diameter smaller than the resolution of the surgical microscope is used to approximate an ideal point light source image. The focal point of the objective lens is at the center of the microsphere as the focal plane position 0, and the original depth of field is set to ±aμm. The objective lens focus is moved up and down ±bμm to sample the corresponding images at different positions, where b>a. After image cropping, background removal, pixel value inversion, and pixel normalization, the corresponding Airy disk image is obtained, that is, the three-dimensional point spread function of the microscope optical system. S3: Build a random depth face database; scale the face depth range to the sampling depth range of the acquired point spread function; Constructing a random depth face database, specifically: constructing a random two-dimensional frequency domain distribution, dividing the amplitude of each frequency component by the total amplitude to obtain the relative amplitude of the frequency component, calculating the power spectrum density of each frequency component, thereby obtaining a two-dimensional power spectrum density, performing a discrete Fourier transform on the two-dimensional power spectrum density, obtaining a simulated artificial random face with random depth and surface roughness, and repeating this process to construct a random depth face database; S4: simulates the partially out-of-focus image of an optical microscope imaging a non-flat sample; randomly selects an image from S1 as the structure to be imaged; randomly selects a surface type from S3 as the surface depth information of the structure to be imaged, and scales the surface type size to be consistent with the data size of the structure to be imaged; according to the corresponding depth of each position of the structure to be imaged, a convolution operation is performed with the point spread function of the corresponding depth in S2 to simulate the microscopic imaging effect of the corresponding position after diffraction of the microscope light path; after convolution operation of all positions, a partially out-of-focus image corresponding to this surface type and the structure to be imaged is obtained, and after adding Poisson distribution and Gaussian distribution noise to the partially out-of-focus image, the final simulated partially out-of-focus image is obtained; S5: Simulate the perfectly focused image of an optical microscope imaging a flat sample; the same image selected in S4 is selected from S1 as the structure to be imaged, and the surface depth corresponding to this structure is regarded as completely flat, so each position of the structure to be imaged is convolved with the point spread function corresponding to the 0 depth in S2, i.e., the focal plane position, to simulate the microscopic imaging effect of the corresponding position after diffraction. After the convolution operation is performed on all positions, a completely flat surface and a fully focused image corresponding to the structure to be imaged are obtained, and after adding Poisson distribution and Gaussian distribution noise to the fully focused image, the final simulated fully focused image is obtained; S6: combining the partially out-of-focus image and the perfectly focused image generated by steps S4 and S5 of the same structure to be imaged as paired data required for model training, and by repeating steps S4 and S5, generating a large amount of paired data for subsequent model training; S7: Model training; The model training is as follows: construct an encoder-decoder structure network model from image to image; use the partially out-of-focus images of the paired data as model input after image preprocessing, use the fully focused images as labels and the model output results to calculate the loss function, and iterate based on the gradient descent method. After training with massive synthetic paired data, the model output results are close to fully focused images, that is, they have the ability to recover from defocus; S8: Model application; The specific application of the model is: input the intraoperative microscopic imaging results of the surgical microscope into the trained model, and the model output result is the focus image after defocus recovery, which realizes the depth of field extension of the surgical microscope.

3. The method for extending the depth of field of an optical surgical microscope according to claim 1, characterized in that: In step S1, the image may be a natural texture image or a high-definition microscopic image.

4. The method for extending the depth of field of an optical surgical microscope according to claim 1, characterized in that: In step S7, the encoder-decoder structure network model can be U-Net or ResU-Net.

Images