Joint optimization method for extending depth of field of wavefront coding microscopy system based on deep learning

Abstract

The application discloses a wavefront coding microscopic system depth of field expansion joint optimization method based on deep learning, and belongs to the field of microscopic medical imaging. The application comprises the following steps: collecting clear focus images for neural network model training and testing; calculating defocus phases according to corresponding optical system parameters, and simulating defocus blur of the clear focus images; using a learnable aberration correction mask matrix, updating data to correct defocus aberration in the training process; calculating a joint point spread function through the defocus correction mask to generate a defocus correction image; using the defocus correction image and the clear focus image pair, and a point spread function quality and image quality evaluation loss function to train a generative model to generate a high-quality optical image. The application optimizes a mask free surface type, introduces an optical physics mechanism to fuse an image similarity evaluation function to form a model gradient back transmission loss function, ensures the effectiveness of wavefront coding aberration correction, and improves the imaging quality and robustness.

Description

TECHNICAL FIELD

[0001] The application relates to the field of medicine, optics and information science, and particularly to a wavefront coding microscopic system depth of field expansion joint optimization method based on deep learning. BACKGROUND

[0002] Microscopic imaging systems have important applications in the fields of biomedical detection, material science and in-vivo observation, but the traditional systems have an inherent contradiction between the depth of field (DOF) and the resolution. High-resolution imaging requires a large numerical aperture (NA), which leads to a significant reduction in the depth of field, and it is difficult to cover the range of three-dimensional structures or dynamic samples (such as flowing cells). Existing depth expansion methods mainly include: optical aperture reduction method: the depth of field is expanded by reducing the relative aperture, but the resolution and light flux are sacrificed, resulting in a decrease in the image signal-to-noise ratio. Mechanical scanning method: multiple imaging and fusion of different focal planes of the sample, but it takes a long time and cannot be used for real-time monitoring of moving samples (such as living cells). Wavefront coding (WFC) technology: a phase mask plate (such as a cubic phase plate) is added to the imaging system aperture, the wavefront of the system is modulated, the point spread function (PSF) remains consistent (i.e. 'off-focus invariance') in the off-focus range, and the image is restored through digital filtering. This method can expand the depth of field to 10-60 times that of the traditional system, and does not need to sacrifice the optical hardware resolution.

[0003] Although wavefront coding has a significant effect on depth extension, its practicality still faces three major bottlenecks: low design freedom of phase plate: the traditional method uses a fixed surface type (such as a cubic phase function), and the parameter optimization depends on the consistency evaluation of the modulation transfer function (MTF), but the MTF consistency cannot be directly related to the visual quality of the final restored image. The optimization space of the phase plate coefficient (such as the cubic term coefficient α) is limited, and the actual aberration of the optical system and the multi-field effect are not considered. Image restoration and optical design are separated: the phase plate design and the restoration algorithm are optimized independently, which leads to low quality of the intermediate coded image (such as loss of low-frequency information, noise sensitivity). Non-blind deconvolution is needed for restoration, but the actual noise amplification effect is significant, especially in low-illumination scenarios (such as fluorescence imaging), and the texture details are seriously lost. The system is not robust enough: the phase plate processing and assembly errors (such as surface tolerance and optical axis offset) will destroy the PSF consistency, and the traditional method lacks an adaptive compensation mechanism for tolerances, resulting in a lower actual extension effect than the theoretical value.

[0004] In recent years, deep learning has been introduced into the field of computational imaging to improve the quality of recovered images: end-to-end restoration networks (such as ResUNet, GAN) can directly generate clear images from encoded images, but the network performance is limited by the quality of the intermediate optical image. Without a phase plate optimized in advance in the optical system, the network is difficult to directly recover the high-frequency details of the image with a serious defocus degree. There are also works (Jin L, Tang Y, Wu Y, et al. Deep learning extended depth-of-field microscope for fast and slide-free histology[J]. Proceedings of the National Academy of Sciences, 2020, 117(52): 33051-33060.) to explore the joint optimization method, but the existing scheme still has defects: the phase plate surface type adopts a parameterized model (such as Zernike polynomial), which lacks freedom; the optimization function lacks physical structure prior, and the image recovery quality and robustness need to be further improved. SUMMARY

[0005] In view of the current lack of effective and convenient large depth of field microscopic imaging method, the problem of insufficient imaging depth of field and wavefront coding optimization, the purpose of the present application is to provide a wavefront coding microscopic system depth of field expansion joint optimization method based on deep learning.

[0006] The purpose of the present application is realized by the following technical scheme: a wavefront coding microscopic system depth of field expansion joint optimization method based on deep learning, comprising the following steps:

[0007] S1: collect original clear focus images and natural images, wherein the original clear focus images include microscopic section images;

[0008] S2: calculate the defocus blur kernel based on the original clear focus images and natural images, combine the Gaussian imaging formula and the required depth range to be expanded, and obtain the defocus blur kernel at equal interval depths in the depth range;

[0009] S3: calculate the defocus aberration correction mask of the learning wavefront modulation based on the defocus blur kernel;

[0010] S4: point spread function calculation of the overall microscopic optical imaging system, according to the Fourier optical theory, combine the defocus blur phase kernel and the defocus correction phase to calculate the point spread function of the microscopic optical imaging system;

[0011] S5: Defocus correction image construction, corresponding defocus correction images are obtained by convolution of point spread function and original clear focus images, forming clear focus-defocus correction image pairs, inputting into image restoration neural network model;

[0012] S6: End-to-end model training, based on the process of steps S3-S5 and image restoration neural network model, an end-to-end training process is constructed, by minimizing the defocus consistency loss function of the corrected defocus point spread function, the center energy concentration loss function of the point spread function, the error between the restored image and the original clear focus image, and the error between the restored image and the defocus correction image after the point spread function, the weight parameters of the learnable defocus aberration correction mask and the neural network model for image restoration are optimized;

[0013] S7: Collecting blurred images after loading defocus aberration correction mask images in actual microscopic optical imaging system and manually refocused image pairs, further fine-tuning the weight parameters of the neural network model.

[0014] Further, the defocus aberration correction mask for learning wavefront modulation is specifically: a matrix similar in size to the pupil is constructed and initialized as a cubic mask, all data in the matrix are set as learnable parameters, and the defocus aberration correction mask for learning wavefront modulation is calculated in the end-to-end training stage.

[0015] Further, the defocus correction image construction includes: after obtaining the corresponding defocus correction images, random Poisson noise and Gaussian noise are applied to the defocus correction images for simulating environmental noise and system noise, forming clear focus-defocus correction image pairs.

[0016] Further, the defocus consistency loss function of the point spread function includes: the point spread function is not sensitive to defocus by constructing a defocus consistency loss function to optimize network model parameters.

[0017] Further, the center energy concentration loss function of the point spread function includes: the network model is optimized by constructing a center energy concentration loss function of the point spread function to maximize the preservation of image information, so as to improve the signal-to-noise ratio of the wavefront regulation system.

[0018] Further, the error between the restored image and the original clear focus image includes: the error between the restored image and the original clear focus image is calculated by constructing an L1 and L2 joint loss function.

[0019] Further, the error between the restored image and the defocus correction image after the point spread function includes: based on the physical cyclic consistency condition, the error between the restored image and the defocus correction image obtained by convolution of the point spread function is calculated by constructing an L1 and L2 joint loss function.

[0020] The application further provides a depth-of-field expansion joint optimization device of a wavefront coding microscopic system based on deep learning, comprising an objective lens, a mirror, a lens, a spatial light modulator and a camera, and is used for realizing the depth-of-field expansion joint optimization method of the wavefront coding microscopic system based on deep learning.

[0021] The application further provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; the memory is used for storing program data, and the processor is used for executing the program data to realize the depth-of-field expansion joint optimization method of the wavefront coding microscopic system based on deep learning.

[0022] The application further provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the program is executed by a processor to realize the depth-of-field expansion joint optimization method of the wavefront coding microscopic system based on deep learning.

[0023] The application has the following beneficial effects: the application provides a depth-of-field expansion joint optimization method of a wavefront coding microscopic system based on deep learning, differential optical modeling and deep learning restoration network are deeply fused, a more free phase mask surface type and a loss function fusing physical prior are designed, global optimization of depth-of-field expansion and imaging quality is realized, and a new paradigm core innovation is provided for high-precision three-dimensional microscopic imaging. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1A schematic diagram of a depth learning-based wavefront coding microscopic system depth-of-field expansion joint optimization method of the present application is provided.

[0026] Figure 2 Each depth PSF graph provided by the embodiment of the present application is provided.

[0027] Figure 3 The overall recovery effect graph comparison graph provided by the embodiment of the present application is provided.

[0028] Figure 4 The optical device schematic diagram provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0030] The embodiment of the present application provides a depth learning-based wavefront coding microscopic system depth-of-field expansion joint optimization method, which specifically includes the following steps:

[0031] S1: Collecting original clear focus images and natural images, wherein the original clear focus images include microscopic section images;

[0032] S2: Calculating defocus blur kernels based on the original clear focus images and natural images, combining the Gaussian imaging formula and the required expanded depth range, to obtain the defocus blur kernels at equal interval depths in the depth range;

[0033] S3: Calculating the defocus aberration correction mask of the learning wavefront modulation based on the defocus blur kernels;

[0034] S4: Point spread function calculation of the overall microscopic optical imaging system, according to the Fourier optical theory combining the defocus blur phase kernel and the defocus correction phase to calculate the point spread function of the microscopic optical imaging system;

[0035] S5: Defocus correction image construction, the corresponding defocus correction image is obtained by convolution of the point spread function and the original clear focus image, forming a clear focus-defocus correction image pair, which is input to the image restoration neural network model;

[0036] S6: End-to-end model training, based on the processes of steps S3-S5, an end-to-end training process is jointly constructed with an image restoration neural network model, by minimizing the defocus consistency loss function of the corrected defocus point spread function, the center energy concentration loss function of the point spread function, and the error between the restored image and the original clear focus image, and the error between the restored image after the point spread function and the defocus corrected image, the weight parameters of the learnable defocus aberration correction mask and the neural network model for image restoration are optimized;

[0037] S7: Collecting the blurred image after loading the defocus aberration correction mask image in the actual microscopic optical imaging system and the manually refocused image pair, and further fine-tuning the weight parameters of the neural network model.

[0038] As shown in Figure 1 , the embodiment of the present application also provides a specific implementation process of a wavefront coding microscopic system depth of field expansion joint optimization method based on deep learning, including a defocus correction training data pair making stage, an end-to-end model training stage, and a model actual system running stage, specifically including the following steps:

[0039] (1) Defocus correction training data pair making stage

[0040] First, collect clear focus images including but not limited to microscopic section images, which contain images with rich texture or structural details, and a series of related open source image data can be used.

[0041] Then, calculate the defocus blur kernel to obtain the defocus blur kernel at each depth, as shown in the defocus phase in Figure 1 .

[0042] Next, defocus aberration correction mask calculation, a matrix with a size similar to the pupil size is constructed and initialized as a cubic mask, all data in the matrix are set as learnable parameters, and the defocus aberration correction mask of wavefront modulation is calculated and learned in the end-to-end training stage, as shown in the learnable mask matrix in Figure 1 .

[0043] Second, the point spread function of the overall system is calculated, and the point spread function of the system is calculated according to the Fourier optical theory combined with the defocus blur phase kernel and the defocus correction phase, as shown in the simulation PSF in Figure 1 .

[0044] Finally, defocus correction image construction, the corresponding defocus correction image is obtained by convolution of the point spread function and the clear focus image, and random Poisson noise and Gaussian noise are applied to it to simulate environmental noise and system noise, and finally a clear focus-defocus correction image pair is formed and sent into the image restoration neural network model.

[0045] (2) End-to-end model training stage

[0046] Based on the defocus correction training data, the defocus aberration correction mask obtained in the manufacturing stage, the clear focus-defocus correction image pair and the image restoration neural network model are jointly constructed to form an end-to-end training process. Through minimizing the defocus consistency of the corrected defocus point spread function, the central energy concentration, the error between the restored image and the original clear image, and the error between the restored image and the defocus correction image, the weight parameters of the learnable defocus aberration correction mask and the neural network model for image restoration are optimized.

[0047] Wherein, the defocus consistency loss function of the point spread function is to ensure that the corrected point spread function remains consistent at different depths, i.e. independent of distance, and the specific formula is as follows:

[0048] ;

[0049] In the formula, Ω represents the entire coordinate set of the image, represents the point spread function image of the i-th distance, represents the point spread function image of the i+1-th distance.

[0050] The central energy concentration loss function of the point spread function is to ensure that the corrected point spread function can reduce the loss of original information and preserve the details of the image and improve the imaging quality as much as possible. Since the central energy needs to be maximized, the optimization loss function is usually to minimize the loss function, so here it is transformed into minimizing the energy of the non-central region, which is equivalent to maximizing the central energy. The specific formula is as follows:

[0051] ;

[0052] In the formula, Circle represents the center of the image as the center and the smaller distance as the radius, which is set according to the actual situation.

[0053] The error between the restored image and the original clear image, the target is to make the image restoration network restore the defocus correction image as much as possible to be consistent with the original clear image, and the L1 loss function and the root mean square error joint loss are adopted.

[0054] ;

[0055] In the formula, N represents the number of sample data for one training; represents the true value, i.e. the original clear focus image; represents the image restored by the network model.

[0056] The recovered image is subjected to an error between a point spread function and an out-of-focus correction image, and a physically guided cyclic consistency loss is used to ensure consistency of conversion.

[0057] ;

[0058] The final loss function is composed of the above four parts, that is:

[0059] ;

[0060] The out-of-focus aberration correction mask and the image recovery network parameters are simultaneously optimized by gradient backpropagation.

[0061] In one embodiment, in order to evaluate the out-of-focus correction effect of the mask, the application calculates the corresponding PSF at different depths, as shown in Figure 2 In the range of 100 microns, the PSF can maintain good consistency, and the energy is distributed in a small central area, which theoretically has good information transmission capability. In order to evaluate the effect of the whole model, the application tests some data that have not been seen in training, as shown in Figure 3 Compared with the method proposed by Jin L et al., the method can better restore the defocused image. From the quantitative result comparison in Table 1, the method achieves good results in PSNR, SSIM and RSME three indicators, and has higher improvement compared with the method of Jin L et al.

[0062] Table 1: Quantitative comparison results

[0063] (3) Model actual system running phase

[0064] Load the trained phase mask into the spatial light modulator of the optical device as shown in Figure 4 for actual out-of-focus control, and input the image captured by the camera into the image recovery network to obtain a clear image, thereby realizing depth of field expansion.

[0065] The application further provides an electronic device, including a memory and a processor, the memory is coupled with the processor; wherein the memory is used for storing program data, and the processor is used for executing the program data to realize the depth learning based wavefront coding microscopic system depth of field expansion joint optimization method.

[0066] The application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the depth learning based wavefront coding microscopic system depth of field expansion joint optimization method.

[0067] The computer readable storage medium can be an internal storage unit of any of the aforementioned data processing capable devices, such as a hard disk or a memory. The computer readable storage medium can also be any of the aforementioned data processing capable devices, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. Further, the computer readable storage medium can also include both an internal storage unit of any of the aforementioned data processing capable devices and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the aforementioned data processing capable devices, and can also be used to temporarily store data that has been output or is to be output.

[0068] The above description is merely illustrative of the embodiments of the present application and is not intended to limit the present application. The present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A deep learning based joint optimization method for extending the depth of field of a wavefront coding microscopy system, characterized in that, The method comprises the following steps: S1: collecting original clear focus images and natural images, wherein the original clear focus images comprise microscopic section images; S2: calculating an out-of-focus blur kernel based on the original clear focus images and the natural images, combining a Gaussian imaging formula and a required extended depth range to obtain the out-of-focus blur kernel at equidistant depths in the depth range; S3: calculating a learning wavefront modulation out-of-focus aberration correction mask based on the out-of-focus blur kernel; S4: calculating a point spread function of the overall microscopic optical imaging system, combining the out-of-focus blur phase kernel and the out-of-focus correction phase to calculate the point spread function of the microscopic optical imaging system according to Fourier optical theory; S5: constructing an out-of-focus correction image, obtaining the corresponding out-of-focus correction image by convolution of the point spread function and the original clear focus image, forming a clear focus-out-of-focus correction image pair, and inputting the clear focus-out-of-focus correction image pair into an image restoration neural network model; S6: training an end-to-end model, constructing an end-to-end training process based on the steps S3-S5 and the image restoration neural network model, and optimizing the learnable out-of-focus aberration correction mask and the weight parameters of the neural network model for image restoration by minimizing the out-of-focus consistency loss function of the corrected out-of-focus point spread function, the center energy concentration loss function of the point spread function, the error between the restored image and the original clear focus image, and the error between the restored image and the out-of-focus correction image after the restored image passes through the point spread function; S7: collecting a blurred image after loading the out-of-focus aberration correction mask image in the actual microscopic optical imaging system and a manually refocused image pair, and further fine-tuning the weight parameters of the neural network model.

2. The deep learning based joint optimization method of depth of field extension for wavefront coding microscopy system according to claim 1, wherein, The calculation of the learning wavefront modulation out-of-focus aberration correction mask specifically comprises: constructing a matrix similar in size to the pupil, initializing the matrix as a cubic mask, setting all data in the matrix as learnable parameters, and calculating the learning wavefront modulation out-of-focus aberration correction mask in the end-to-end training stage.

3. The deep learning based joint optimization method of depth of field extension for wavefront coding microscopy system according to claim 1, wherein, The out-of-focus correction image construction comprises: after obtaining the corresponding out-of-focus correction image, applying random Poisson noise and Gaussian noise to the out-of-focus correction image to simulate environmental noise and system noise, and forming a clear focus-out-of-focus correction image pair.

4. The deep learning based joint optimization method of depth of field extension for wavefront coding microscopy system according to claim 1, wherein, The out-of-focus consistency loss function of the point spread function comprises: optimizing the network model parameters by constructing the out-of-focus consistency loss function to make the point spread function insensitive to out-of-focus.

5. The deep learning based joint optimization method of depth of field extension for wavefront coding microscopy system according to claim 1, wherein, The center energy concentration loss function of the point spread function comprises: optimizing the network model by constructing the center energy concentration loss function of the point spread function to maximize the preservation of image information and improve the signal-to-noise ratio of the wavefront regulation system.

6. The deep learning based joint optimization method of depth of field extension for wavefront coding microscopy system according to claim 1, wherein, The error between the restored image and the original clear focus image comprises: calculating the error between the restored image and the original clear focus image by constructing an L1 and L2 joint loss function.

7. The deep learning based joint optimization method of depth of field extension for wavefront coding microscopy system according to claim 1, wherein, The error between the restored image and the out-of-focus correction image after the restored image passes through the point spread function comprises: calculating the error between the restored image and the out-of-focus correction image obtained by convolution of the point spread function based on the physical cyclic consistency condition by constructing an L1 and L2 joint loss function.

8. A joint optimization device for depth-of-field extension in a wavefront-coded microscopy system based on deep learning, characterized in that, An objective lens, a mirror, a lens, a spatial light modulator and a camera are included to implement the deep learning based wavefront coding microscopic system depth of field expansion joint optimization method as claimed in any one of claims 1-7.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled with the processor; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the deep learning based wavefront coding microscopic system depth of field expansion joint optimization method as claimed in any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the deep learning based wavefront coding microscopic system depth of field expansion joint optimization method as claimed in any one of claims 1-8.

Images