Wavefront coding microscopic system depth-of-field expansion joint optimization method based on deep learning

Abstract

The invention discloses a wavefront coding microscopic system depth-of-field expansion joint optimization method based on deep learning, and belongs to the field of microscopic medical imaging, and the method comprises the steps: collecting a clear focusing image for neural network model training and testing; a defocusing phase is calculated according to corresponding optical system parameters, and defocusing blurring simulation is carried out on the clear focusing image; correcting a mask matrix by utilizing learnable aberration, and updating data in a training process to correct out-of-focus aberration; calculating a joint point spread function through the out-of-focus correction mask, and generating an out-of-focus correction image; and training a generative model by using a pair of the defocus correction image and the clear focusing image and a point spread function quality and image quality evaluation loss function, and recovering the optical image with high quality. According to the method, the free surface type of the mask is optimized, an optical physical mechanism is introduced to be fused with an image similarity evaluation function to form a model gradient return loss function, the aberration correction effectiveness of wavefront coding is ensured, and the imaging quality and robustness are improved.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of medicine, optics and information, and in particular to a joint optimization method for depth of field extension of a wavefront coding microscopy system based on deep learning. Background Art

[0002] Microscopic imaging systems have important applications in biomedical testing, materials science, and in vivo observation. However, conventional systems suffer from an inherent conflict between depth of field (DOF) and resolution. High-resolution imaging requires a large numerical aperture (NA), which significantly reduces the DOF, making it difficult to capture three-dimensional sample structures or dynamic samples (such as flowing cells). Existing methods for extending the DOF include: Optical aperture reduction: This method increases the DOF by reducing the relative aperture, but this sacrifices resolution and light throughput, resulting in a decrease in the image signal-to-noise ratio. Mechanical scanning: This method images the sample multiple times at different focal planes and fuses them, but this is time-consuming and unsuitable for real-time monitoring of moving samples (such as living cells). Wavefront coding (WFC) technology: This technique modulates the system wavefront by inserting a phase mask (such as a cubic phase plate) at the aperture of the imaging system, ensuring that the point spread function (PSF) remains consistent across the defocus range (i.e., "defocus invariance"). The image is then restored through digital filtering. This method can extend the DOF by 10–60 times that of conventional systems without sacrificing optical hardware resolution.

[0003] Although wavefront coding has demonstrated significant benefits in extending depth of field, its practical application faces three major bottlenecks: Phase plate design has limited freedom: Traditional methods employ a fixed surface shape (e.g., a cubic phase function), and parameter optimization relies on modulation transfer function (MTF) consistency evaluation. However, MTF consistency cannot directly correlate with the visual quality of the final restored image. The optimization space for phase plate coefficients (e.g., the cubic term coefficient α) is limited, and actual optical system aberrations and multi-field effects are not considered. Image restoration is decoupled from optical design: Phase plate design and restoration algorithms are optimized independently, resulting in poor intermediate encoded image quality (e.g., loss of low-frequency information and noise sensitivity). Restoration relies on non-blind deconvolution, but this can significantly amplify noise, particularly in low-light scenarios (e.g., fluorescence imaging), where texture detail is severely lost. Insufficient system robustness: Phase plate machining and assembly errors (e.g., surface tolerances and optical axis offset) can disrupt PSF consistency. Traditional methods lack adaptive compensation mechanisms for these tolerances, resulting in actual extension performance falling short of theoretical values.

[0004] In recent years, deep learning has been introduced into the field of computational imaging to improve the quality of restored images: end-to-end restoration networks (such as ResUNet and GAN) can directly generate clear images from encoded images, but the network performance is limited by the quality of the intermediate optical image. Without the phase plate optimized in advance in the optical system, it is difficult for the network to directly restore the high-frequency details of images with severe defocus. There are also works (Jin L, Tang Y, Wu Y, et al. Deep learning extended depth-of-field microscope for fast and slide-free histology[J]. Proceedings of the National Academy of Sciences, 2020, 117(52): 33051-33060.) to explore joint optimization methods, but existing solutions still have defects: the phase plate surface uses a parameterized model (such as Zernike polynomials) with insufficient degrees of freedom; the optimization function lacks physical structure priors, and the image restoration quality and robustness need to be further improved. Summary of the Invention

[0005] In view of the current lack of effective and convenient large-depth-of-field microscopic imaging methods, insufficient imaging depth of field, and difficulty in wavefront coding optimization, the purpose of the present invention is to provide a deep learning-based joint optimization method for depth-of-field expansion of wavefront coding microscopic systems.

[0006] The objective of the present invention is achieved through the following technical solution: a joint optimization method for depth of field extension of a wavefront coding microscopy system based on deep learning, comprising the following steps:

[0007] S1: collecting original clear-focus images and natural images, wherein the original clear-focus images include microscopic section images;

[0008] S2: calculating a defocus blur kernel based on the original clearly focused image and the natural image, and obtaining a defocus blur kernel at equally spaced depths within the depth range by combining a Gaussian imaging formula and the desired extended depth range;

[0009] S3: calculating a defocus aberration correction mask of a learning wavefront modulation based on the defocus blur kernel;

[0010] S4: Calculation of the point spread function of the overall microscopic optical imaging system. The point spread function of the microscopic optical imaging system is calculated based on Fourier optics theory combined with the defocus blur phase kernel and defocus correction phase.

[0011] S5: Defocus correction image construction, by convolving the point spread function with the original clear focus image to obtain the corresponding defocus correction image, forming a clear focus-defocus correction image pair, which is input into the image restoration neural network model;

[0012] S6: end-to-end model training, based on the process of steps S3-S5 and the image restoration neural network model, jointly constructing an end-to-end training process, by minimizing the defocus consistency loss function of the corrected defocus point spread function, the central energy concentration loss function of the point spread function, the error between the restored image and the original clear focused image, and the error between the restored image through the point spread function and the defocus corrected image, thereby optimizing the learnable defocus aberration correction mask and the weight parameters of the neural network model for image restoration;

[0013] S7: Collect blurred images and manually refocused image pairs after loading the defocus aberration correction mask image in the actual microscopic optical imaging system to further fine-tune the weight parameters of the neural network model.

[0014] Furthermore, the calculation of the defocus aberration correction mask for the learning wavefront modulation is specifically as follows: a matrix close to the size of the pupil is constructed and initialized as a cubic mask, all data in the matrix are set as learnable parameters, and the defocus aberration correction mask for the learning wavefront modulation is calculated in the end-to-end training stage.

[0015] Furthermore, the defocus correction image construction includes: after obtaining the corresponding defocus correction image, applying random Poisson noise and Gaussian noise to the defocus correction image to simulate environmental noise and system noise to form a clear focus-defocus correction image pair.

[0016] Furthermore, the defocus consistency loss function of the point spread function includes: optimizing network model parameters by constructing a defocus consistency loss function to achieve point spread function insensitivity to defocus.

[0017] Furthermore, the central energy concentration loss function of the point spread function includes: retaining image information to the greatest extent by constructing a point spread function central energy concentration loss function optimization network model to improve the signal-to-noise ratio of the wavefront control system.

[0018] Furthermore, calculating the error between the restored image and the original clear and focused image includes: calculating the error between the restored image and the original clear and focused image by constructing an L1 and L2 joint loss function.

[0019] Furthermore, the error between the restored image after the point spread function and the defocus corrected image includes: the error between the restored image and the defocus corrected image obtained by convolving the restored image and the point spread function by constructing an L1 and L2 joint loss function under physical cycle consistency conditions.

[0020] The present invention also provides a deep learning-based joint optimization device for depth of field extension of a wavefront coding microscopy system, comprising an objective lens, a reflector, a lens, a spatial light modulator, and a camera, for implementing the deep learning-based joint optimization method for depth of field extension of a wavefront coding microscopy system.

[0021] The present invention also provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the deep learning-based joint optimization method for depth of field extension of a wavefront coding microscope system.

[0022] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the joint optimization method for depth of field extension of a wavefront coding microscope system based on deep learning.

[0023] The beneficial effects of the present invention are: the present invention proposes a joint optimization method for depth of field extension of a wavefront coding microscopy system based on deep learning, and through the deep fusion of differentiable optical modeling and deep learning restoration network, designs a more free phase plate surface shape and integrates the loss function of physical priors, thereby achieving global optimization of depth of field extension and imaging quality, and providing a new paradigm for high-precision three-dimensional microscopy imaging. The core innovation lies in: free surface shape of the phase mask: with the cubic mask as the initialization state, all surface shape data can be learned, rather than the currently commonly used Zenike surface fitting method, thereby further improving the degree of freedom of the phase mask surface shape and improving the efficiency of wavefront coding. End-to-end collaborative optimization of the phase mask plate and neural network: A differentiable forward model including optical modulation and model restoration is constructed, so that its gradient can be back-propagated to the phase plate surface parameters. At the same time, the point spread function defocus consistency and center energy maximization loss function are introduced according to the optical characteristics. In the model optimization stage, the L1 and L2 joint optimization loss functions are proposed to optimize the final imaging quality. Finally, through the physically guided cycle consistency structure, the final restored image is convolved with the original optically debugged image for similarity comparison, which improves the stability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1Schematic diagram of the joint optimization method for depth of field extension of a wavefront coding microscopy system based on deep learning of the present invention;

[0026] Figure 2 PSF diagrams of various depths provided by the embodiments of the present invention;

[0027] Figure 3 A comparison diagram of the overall recovery effect provided by an embodiment of the present invention;

[0028] Figure 4 A schematic diagram of an optical device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0030] The embodiment of the present invention provides a joint optimization method for depth of field extension of a wavefront coding microscopy system based on deep learning, which specifically includes the following steps:

[0031] S1: collecting original clear-focus images and natural images, wherein the original clear-focus images include microscopic section images;

[0032] S2: calculating a defocus blur kernel based on the original clearly focused image and the natural image, and obtaining a defocus blur kernel at equally spaced depths within the depth range by combining a Gaussian imaging formula and the desired extended depth range;

[0033] S3: calculating a defocus aberration correction mask of a learning wavefront modulation based on the defocus blur kernel;

[0034] S4: Calculation of the point spread function of the overall microscopic optical imaging system. The point spread function of the microscopic optical imaging system is calculated based on Fourier optics theory combined with the defocus blur phase kernel and defocus correction phase.

[0035] S5: Defocus correction image construction, by convolving the point spread function with the original clear focus image to obtain the corresponding defocus correction image, forming a clear focus-defocus correction image pair, which is input into the image restoration neural network model;

[0036] S6: end-to-end model training, based on the process of steps S3-S5 and the image restoration neural network model, jointly constructing an end-to-end training process, by minimizing the defocus consistency loss function of the corrected defocus point spread function, the central energy concentration loss function of the point spread function, the error between the restored image and the original clear focused image, and the error between the restored image through the point spread function and the defocus corrected image, thereby optimizing the learnable defocus aberration correction mask and the weight parameters of the neural network model for image restoration;

[0037] S7: Collect blurred images and manually refocused image pairs after loading the defocus aberration correction mask image in the actual microscopic optical imaging system to further fine-tune the weight parameters of the neural network model.

[0038] like Figure 1 As shown, the embodiment of the present invention also provides a specific implementation process of a joint optimization method for depth of field extension of a wavefront coding microscopy system based on deep learning, including the preparation of defocus correction training data pairs, end-to-end model training and actual system operation of the model, specifically including the following steps:

[0039] (1) Defocus correction training data generation stage

[0040] First, collect clearly focused images, including but not limited to microscopic section images, which contain rich texture or structural details, and use a series of relevant open source image data.

[0041] Then, the defocus blur kernel is calculated to obtain the defocus blur kernel at each depth, such as Figure 1 The defocused phase is shown in .

[0042] Next, the defocus aberration correction mask is calculated. A matrix of approximately the same size as the pupil is constructed and initialized as a cubic mask. All data in the matrix are set as learnable parameters. The defocus aberration correction mask of the learning wavefront modulation is calculated in the end-to-end training phase, as shown in Figure 1 The learnable mask matrix in .

[0043] Secondly, the point spread function of the whole system is calculated. According to Fourier optics theory, the point spread function of the system is obtained by combining the defocus blur phase kernel and the defocus correction phase, as shown in Figure 1 The simulated PSF in .

[0044] Finally, the defocus correction image is constructed by convolving the point spread function with the clear focus image to obtain the corresponding defocus correction image, and random Poisson noise and Gaussian noise are applied to it to simulate environmental noise and system noise, finally forming a clear focus-defocus correction image pair and sending it to the image restoration neural network model.

[0045] (2) End-to-end model training phase

[0046] An end-to-end training process is constructed based on the defocus aberration correction mask obtained in the production phase of the defocus correction training data, the clear focus-defocus correction image pair and the image restoration neural network model. The learnable defocus aberration correction mask and the weight parameters of the neural network model for image restoration are optimized by minimizing the defocus consistency and central energy centralization of the corrected defocus point spread function, the error between the restored image and the original clear image, and the error between the restored image and the defocus corrected image through the point spread function.

[0047] Among them, the defocus consistency loss function of the point spread function is to ensure that the corrected point spread function remains consistent at different depths, that is, it is independent of the distance. The specific formula is as follows:

[0048] ;

[0049] Where Ω represents the entire The coordinate set of the image, represents the point spread function image of the i-th distance, Represents the point spread function image at the i+1th distance.

[0050] The central energy concentration loss function of the point spread function is to ensure that the corrected point spread function can reduce the loss of original information, preserve the details of the image as much as possible and improve the imaging quality. Since it is necessary to maximize the central energy, the optimization loss function is often to minimize the loss function. Therefore, it is converted here to minimizing the energy of the non-central area, which is disguised as maximizing the central capacity. The specific formula is as follows:

[0051] ;

[0052] In the formula, Circle represents a circle with the center of the image as the center and the smallest distance as the radius. This value should be set according to the actual situation.

[0053] The error between the restored image and the original clear image. The goal is to make the image restoration network restore the defocused corrected image as consistent as possible with the original clear image. The L1 loss function and the root mean square error loss are used.

[0054] ;

[0055] In the formula, N represents the number of sample data for one training; represents the ground truth, i.e. the original sharp focused image; Represents an image recovered by the network model.

[0056] The restored image is subjected to the error between the point spread function and the defocus-corrected image, and a physics-guided cycle consistency loss is adopted to ensure the consistency of the transformation.

[0057] ;

[0058] The final loss function consists of the above four parts, namely:

[0059] ;

[0060] By gradient backpropagation, the defocus aberration correction mask and image restoration network parameters are optimized simultaneously.

[0061] In one embodiment, in order to evaluate the defocus correction effect of the mask, the present invention calculates the corresponding PSF at different depths, such as Figure 2 As shown in Figure 1, within the range of 100 microns, the PSF can maintain good consistency, and the energy is distributed in a small central area, which theoretically has good information transmission capabilities. In order to evaluate the effect of the entire model, the present invention tested some data that had not been seen in training, such as Figure 3 As shown in Table 1, our method can better restore defocused images than the method proposed by Jin L et al. From the quantitative results comparison in Table 1, we can see that our method achieves good results in terms of PSNR, SSIM, and RSME, and has a significant improvement over the method proposed by Jin L et al.

[0062] Table 1: Quantitative comparison results

[0063] (3) Model actual system operation stage

[0064] Load the trained phase mask into Figure 4 The spatial light modulator in the optical device shown is used for actual defocus control, and the image captured by the camera is input into the image restoration network to obtain a clear image and achieve depth of field extension.

[0065] The present invention also provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the deep learning-based joint optimization method for depth of field extension of a wavefront coding microscope system.

[0066] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the joint optimization method for depth of field extension of a wavefront coding microscope system based on deep learning.

[0067] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0068] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A joint optimization method for depth of field extension of wavefront coding microscopy system based on deep learning, characterized by: The steps include: S1: collecting original clear-focus images and natural images, wherein the original clear-focus images include microscopic section images; S2: calculating a defocus blur kernel based on the original clearly focused image and the natural image, and obtaining a defocus blur kernel at equally spaced depths within the depth range by combining a Gaussian imaging formula and the desired extended depth range; S3: calculating a defocus aberration correction mask of a learning wavefront modulation based on the defocus blur kernel; S4: Calculation of the point spread function of the overall microscopic optical imaging system. The point spread function of the microscopic optical imaging system is calculated based on Fourier optics theory combined with the defocus blur phase kernel and defocus correction phase. S5: Defocus correction image construction, by convolving the point spread function with the original clear focus image to obtain the corresponding defocus correction image, forming a clear focus-defocus correction image pair, which is input into the image restoration neural network model; S6: end-to-end model training, based on the process of steps S3-S5 and the image restoration neural network model, jointly constructing an end-to-end training process, by minimizing the defocus consistency loss function of the corrected defocus point spread function, the central energy concentration loss function of the point spread function, the error between the restored image and the original clear focused image, and the error between the restored image through the point spread function and the defocus corrected image, thereby optimizing the learnable defocus aberration correction mask and the weight parameters of the neural network model for image restoration; S7: Collect blurred images and manually refocused image pairs after loading the defocus aberration correction mask image in the actual microscopic optical imaging system to further fine-tune the weight parameters of the neural network model.

2. The method for joint optimization of depth of field extension of wavefront coding microscopy system based on deep learning according to claim 1, characterized in that: The calculation of the defocus aberration correction mask for the learning wavefront modulation is specifically as follows: a matrix with a size close to that of the pupil is constructed and initialized as a cubic mask, all data in the matrix are set as learnable parameters, and the defocus aberration correction mask for the learning wavefront modulation is calculated in the end-to-end training stage.

3. The method for joint optimization of depth of field extension of wavefront coding microscopy system based on deep learning according to claim 1, characterized in that: The defocus correction image construction includes: after obtaining the corresponding defocus correction image, applying random Poisson noise and Gaussian noise to the defocus correction image to simulate environmental noise and system noise to form a clear focus-defocus correction image pair.

4. The method for joint optimization of depth of field extension of wavefront coding microscopy system based on deep learning according to claim 1, characterized in that: The defocus consistency loss function of the point spread function includes: optimizing network model parameters by constructing a defocus consistency loss function to achieve point spread function insensitivity to defocus.

5. The method for joint optimization of depth of field extension of wavefront coding microscopy system based on deep learning according to claim 1, characterized in that: The central energy concentration loss function of the point spread function includes: retaining image information to the greatest extent by constructing a point spread function central energy concentration loss function optimization network model to improve the signal-to-noise ratio of the wavefront control system.

6. The method for joint optimization of depth of field extension of wavefront coding microscopy system based on deep learning according to claim 1, characterized in that: The error between the restored image and the original clear and focused image comprises: calculating the error between the restored image and the original clear and focused image by constructing an L1 and L2 joint loss function.

7. The method for joint optimization of depth of field extension of wavefront coding microscopy system based on deep learning according to claim 1, characterized in that: The error between the restored image after the point spread function and the defocus corrected image includes: the error between the restored image and the defocus corrected image obtained by convolving the restored image and the point spread function by constructing an L1 and L2 joint loss function under physical cycle consistency conditions.

8. A deep learning-based joint optimization device for depth of field extension of a wavefront coding microscopy system, characterized in that: The invention comprises an objective lens, a reflector, a lens, a spatial light modulator and a camera, and is used to implement a joint optimization method for depth of field extension of a wavefront coding microscopy system based on deep learning as described in any one of claims 1 to 7.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the deep learning-based wavefront coding microscopy system depth of field extension joint optimization method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for joint optimization of depth of field extension of a wavefront coding microscopy system based on deep learning is implemented.

Images