Method and system for extending depth of field of underwater in-situ microscopic imaging device based on deep learning

Abstract

The present invention discloses a method and system for extending the depth of field of an underwater in-situ microscopic imager based on deep learning. The method comprises the following steps: S1, modeling the optical system to be extended as a differentiable mathematical model; S2, adding a phase mask represented by a discretized surface shape to the differentiable mathematical model; S3, obtaining the point spread function of the imaging system from the mathematical model with the phase mask added; convolving the point spread function with the image and simulating sensor sampling to obtain an image captured by the sensor; S4, inputting the image captured by the sensor into a neural network for image restoration; S5, optimizing the phase mask surface shape and the neural network parameters, with the imaging quality within the final large depth of field range as the optimization target. The present invention can improve the depth of field of the microscopic imager while also enhancing the imaging quality within the extended depth of field range.

Description

Technical Field

[0001] The present invention relates to the field of plankton monitoring technology, and in particular to a method and system for extending the depth of field of an underwater in-situ microscopic imager based on deep learning. Background Art

[0002] The abundance and composition of plankton are closely linked to changes in the local environment. Underwater in situ microscopy based on optical imaging technology is the most efficient monitoring method. However, due to the relatively large aperture of in situ plankton microscopy systems, the depth of field is small, making it difficult to achieve both high resolution and a high depth of field. Plankton can easily escape the depth of field, reducing monitoring efficiency.

[0003] To address the limited depth of field of underwater microscopic imagers, there are three main methods for extending the depth of field. The first is to reduce the relative aperture of the imaging system, based on the principle that the depth of field is inversely proportional to the square of its relative aperture. However, this method reduces the system's resolution and image quality. The second is to image an area beyond the system's depth of field multiple times and then register and fuse the images collected at different depths to obtain a final image with a large depth of field. However, this method requires multiple focus imaging cycles and is unsuitable for dynamic in situ monitoring of underwater plankton. The third is wavefront coding technology, which phase-modulates the light wave using a designed phase mask. This allows the system to obtain a highly consistent point spread function over a large area, and then restores the image with a large depth of field using restoration techniques. However, traditional wavefront coding technology uses a fixed surface representation for surface design, resulting in limited freedom in surface design. The phase mask design optimizes the modulation transfer function (MTF) for consistency, which cannot guarantee the quality of the final restored image. Moreover, the surface design and image restoration algorithm are independent and lack correlation. Summary of the Invention

[0004] The purpose of the present invention is to solve the technical problem of low imaging quality after the depth of field of a microscopic imager is extended, and to propose a method and system for extending the depth of field of an underwater in-situ microscopic imager based on deep learning.

[0005] The technical problem of the present invention is solved by the following technical solutions:

[0006] The present invention discloses a method for extending the depth of field of an underwater in-situ microscopic imager based on deep learning, comprising the following steps:

[0007] S1. Modeling the optical system to be extended in depth of field as a differentiable mathematical model;

[0008] S2. Adding a phase mask template using a discretized surface expression into the differentiable mathematical model;

[0009] S3. Obtaining a point spread function of the imaging system using a mathematical model incorporating a phase mask; convolving the point spread function with the image and simulating sensor sampling to obtain an image captured by the sensor;

[0010] S4, inputting the image collected by the sensor into the neural network for image restoration;

[0011] S5. Taking the imaging quality within the final large depth of field range as the optimization goal, the phase mask template shape and neural network parameters are optimized.

[0012] In some embodiments, step S1 specifically includes:

[0013] S11. According to Fourier optics, the point spread function of the image plane (x2, y2) is discretized and modeled as the square of the Fourier transform of the generalized pupil function:

[0014]

[0015] in represents the two-dimensional Fourier transform process, P(x1,y1) is the generalized pupil function, which can be expressed as:

[0016]

[0017] Among them, (x1, y1) represents each position of the pupil plane, A(x1, y1) represents the amplitude of each position of the pupil plane, φ(x1, y1) represents the phase of each position of the pupil plane, and e i Represents a complex exponential with a natural base;

[0018] S12. Model the phase change caused by defocus as:

[0019]

[0020] in represents a number, λ represents the wavelength of light, z and z0 represent the defocus distance and focus distance of the imaging system respectively, Represents the square of the distance from the coordinate point (x1, y1) on the pupil plane to the center of the pupil plane.

[0021] In some embodiments, step S2 specifically includes: adding a phase mask to the pupil plane, that is, using an additional phase term φ M (x1,y1) to modulate the generalized pupil function:

[0022]

[0023] Wherein, h(x1,y1) is the thickness of each position of the phase mask, and n is the refractive index of the phase mask.

[0024] The phase term in the modulated generalized pupil function is:

[0025] φ(x1,y1,z)=φ OOF (x1,y1,z)+φ M (x1,y1).

[0026] In some embodiments, step S3 specifically includes:

[0027] S31. Substitute the phase term in the modulated generalized pupil function into the point spread function calculation formula to calculate the PSF (x2, y2, z) of the optical system with the phase mask added at each position z0, z1, ... within the depth of field range;

[0028] S32. Use the training image I0(x,y) to convolve with the PSF(x2,y2,z) at different positions of the depth of field to be expanded obtained in the differentiable mathematical model, that is, simulate the imaging image I(x2,y2,z) at different depth of field positions collected by the sensor plane:

[0029]

[0030] Among them, η~N(0,σ 2 ) represents zero-mean Gaussian noise with a standard deviation of σ, which represents the noise when the sensor receives the signal. Represents convolution calculation.

[0031] S33. Calculate the pixel size s in the image I(x2, y2, z) based on the optical system parameters pixel for:

[0032]

[0033] Where f is the focal length of the imaging system and d is the diameter of the pupil plane;

[0034] S34 . Sampling is performed according to the image pixel size and the sensor pixel size to obtain an image S(I) that matches the sensor pixel size.

[0035] In some embodiments, in step S4, the image restoration includes the following steps:

[0036] S41. Each layer of the decoder in the neural network performs two convolutions with a kernel size of 3*3 on the image, and then performs a pooling layer operation on the obtained feature map;

[0037] S42, after four downsamplings, enters the decoder part, and the image also undergoes two convolutions with a convolution kernel size of 3*3, and then upsamples. At the same time, the final feature map obtained by the decoder of the previous layer is connected with the upsampling result;

[0038] S43. After completing the four upsamplings, the image undergoes two more convolutions with a convolution kernel size of 3*3, and then undergoes a convolution with a convolution kernel size of 1*1 and is added to the input image to obtain the final output image.

[0039] In some embodiments, in step S4, the neural network is a ResUnet neural network.

[0040] In some embodiments, step S5 specifically includes:

[0041] S51. The root mean square error between the final output image after image restoration and the training image I0(x,y) is used as the loss function of the system:

[0042]

[0043] Among them, m and n represent the number of pixels of the length and width of the image;

[0044] S52. Optimize the surface function h(x1, y1) of the phase mask and the parameters in the neural network by using the gradient descent method.

[0045] In some embodiments, step S52 specifically includes: cyclically updating the surface parameters of the phase mask and the parameters of the neural network by a gradient descent method until the number of cycles reaches a set threshold, and then stopping the updating.

[0046] The present invention also discloses a depth of field extension system for an underwater in-situ microscope based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods described above when executing the computer program.

[0047] The present invention further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above items is implemented.

[0048] The beneficial effects of the present invention compared with the prior art include:

[0049] The present invention provides a method for extending the depth of field of underwater in-situ microscopic imaging instruments based on deep learning. By employing a phase mask represented by a discretized surface shape, the method increases the degree of freedom in the design of the phase mask's surface shape. Furthermore, a neural network is employed to restore the captured image, thereby improving the imaging quality after the extended depth of field. This method combines the mathematical model of the optical system with the neural network to construct a differentiable deep learning model. The model optimizes the phase mask's surface shape and the parameters of the neural network, targeting the imaging quality within the final extended depth of field. This method improves both the depth of field of the microscopic imaging instrument and the imaging quality within the extended depth of field. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of a method for extending the depth of field of an underwater in-situ microscopic imaging device according to an embodiment of the present invention.

[0051] Figure 2 Schematic diagram of the neural network structure in an embodiment of the present invention.

[0052] Figure 3 This is a flow chart of a joint optimization method for an optical system and image restoration in an embodiment of the present invention.

[0053] Figures 4a to 4d Schematic diagram of the phase mask profile change during the training process of an embodiment of the present invention.

[0054] Figures 5a to 5c 3 is a schematic diagram of the point spread function at different defocus positions after optimization according to an embodiment of the present invention.

[0055] Figures 6a to 6c 3 is an image showing the imaging effects at different defocus positions after optimization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further described below with reference to the accompanying drawings and in combination with preferred embodiments. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0057] It should be noted that the directional terms such as left, right, up, down, top, and bottom in this embodiment are merely relative concepts, or are based on the normal use status of the product, and should not be considered as restrictive.

[0058] The core of the embodiment of the present invention is to build a differentiable mathematical model, that is, the model can optimize the calculation of the entire model through the gradient descent method in deep learning.

[0059] like Figure 1 As shown, the method for extending the depth of field of an underwater in-situ microscopic imager based on deep learning provided by an embodiment of the present invention includes the following steps:

[0060] S1. Model the optical system to be extended in depth of field as a differentiable mathematical model.

[0061] Specifically, according to Fourier optics, the point spread function (PSF) representing the image plane (x2, y2) is discretized and modeled as the square of the Fourier transform of the generalized pupil function:

[0062]

[0063] in represents the two-dimensional Fourier transform process, P(x1,y1) is the generalized pupil function, which can be expressed as:

[0064]

[0065] Among them, (x1, y1) represents each position of the pupil plane, A(x1, y1) represents the amplitude of each position of the pupil plane, φ(x1, y1) represents the phase of each position of the pupil plane, and e i Represents a complex exponential with a natural base;

[0066] When extending the depth of field of an optical system, the traditional imaging system can be considered as an aberration-free system. Therefore, when extending the depth of field based on it, only the defocus aberration caused by leaving the original depth of field range is considered. The phase change caused by defocus can be modeled as:

[0067]

[0068] in represents a number, λ represents the wavelength of light, z and z0 represent the defocus distance and focus distance of the imaging system respectively, Represents the square of the distance from the coordinate point (x1, y1) on the pupil plane to the center of the pupil plane.

[0069] S2. Add a phase mask template using a discretized surface expression into the differentiable mathematical model.

[0070] Specifically, the wavefront coding technology adds a phase mask with a convex structure on the pupil plane, which precisely modulates the phase of the light wave through the convex structure, so that the PSF of the imaging system in a large depth of field range can obtain defocus invariance. Adding a phase mask to the pupil plane means using an additional phase term φ M (x1,y1) to modulate the generalized pupil function:

[0071]

[0072] Wherein, h(x1,y1) is the thickness of each position of the phase mask, and n is the refractive index of the phase mask.

[0073] The phase term in the modulated generalized pupil function is:

[0074] φ(x1,y1,z)=φ OOF (x1,y1,z)+φ M (x1,y1).

[0075] S3. Obtaining a point spread function of the imaging system from a mathematical model incorporating a phase mask; convolving the point spread function with the image and simulating sensor sampling to obtain an image captured by the sensor.

[0076] Specifically, the phase term in the modulated generalized pupil function is substituted into the point spread function calculation formula, based on which the PSF (x2, y2, z) of the optical system with the phase mask added at each position z0, z1, ... within the depth of field can be calculated.

[0077] Because the PSF is equivalent to the imaging effect of the impulse response of the object plane on the image plane after passing through the optical system, the training image I0(x,y) is convolved with the PSF(x2,y2,z) at different positions of the depth of field to be expanded obtained in the differentiable mathematical model, which can simulate the imaging image I(x2,y2,z) at different depths of field positions collected by the sensor plane:

[0078]

[0079] Among them, η~N(0,σ 2 ) represents zero-mean Gaussian noise with a standard deviation of σ, which represents the noise when the sensor receives the signal. Represents convolution calculation.

[0080] Because the neural network restoration scheme operates on pixels in the spatial domain, in order for the restoration scheme to be used in the actual imaging system, it is necessary to consider that the simulation system matches the actual sensor pixel size and calculate the pixel size s in the image I(x2, y2, z) according to the optical system parameters. pixel for:

[0081]

[0082] Where f is the focal length of the imaging system and d is the diameter of the pupil plane. The pixel sizes of the two are then sampled to obtain an image S(I) that matches the sensor pixel size.

[0083] S4. Input the image collected by the sensor into the neural network for image restoration.

[0084] Specifically, the obtained phase mask-encoded images at different positions are input into the image restoration model, that is, the image restoration neural network (ResUnet). The encoded blurred image is restored to a clear image through the neural network (ResUnet). The left half of ResUnet is the decoder. Each layer contains two convolutions, and the convolution kernel size is 3*3. The image size remains unchanged after convolution. After each layer completes two convolutions, the obtained feature map is subjected to a pooling layer operation; after four downsamplings, it enters the decoder part, and the image also undergoes two convolutions with a convolution kernel size of 3*3, and then upsampled. At the same time, the final feature map obtained by the decoder of the previous layer is connected with the upsampling result; after completing four upsamplings, the image undergoes two convolutions with a convolution kernel size of 3*3, and then undergoes a convolution with a convolution kernel size of 1*1 and is added to the input image to obtain the final output image. The structural diagram of the image restoration neural network ResUnet is shown as follows. Figure 2 shown.

[0085] S5. Taking the imaging quality within the final large depth of field range as the optimization goal, the phase mask template shape and neural network parameters are optimized.

[0086] Specifically, in order to ensure the imaging quality of the final depth of field extension system, the root mean square error between the final output image after image restoration and the training image I0(x,y) is used as the loss function of the system:

[0087]

[0088] Among them, m and n represent the number of pixels in the length and width of the image.

[0089] The surface function h(x1,y1) of the phase mask and the parameters of the image restoration neural network ResUnet are optimized by the gradient descent method.

[0090] like Figure 3 As shown, in some embodiments, the surface shape of the phase mask is first initialized, and the surface shape is added to a differentiable mathematical model of the optical system. The point spread function of each position is calculated according to the model, and the point spread function is convolved with the training image and noise is added. Then, it is downsampled to match the sensor pixels, and the obtained sensor image is input into the neural network for restoration. After that, it is optimized by deep learning, that is, the loss function of the restored image and the training image is calculated, the loss function is derived, and the surface parameters of the phase mask and the parameters of the neural network are cyclically updated by the gradient descent method until the number of cycles reaches a set threshold, and then the update is stopped.

[0091] The present invention also discloses a depth of field extension system for an underwater in-situ microscope based on deep learning, which includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, it implements any of the methods described above.

[0092] The present invention further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above methods is implemented.

[0093] The depth of field extension method proposed in the embodiment of the present invention is used in a plankton microscopic imaging instrument, and the depth of field extension effect obtained is shown in Table 1.

[0094] Table 1 Comparison of the effects of traditional wavefront coding and joint model

[0095]

[0096] The experimental settings are as follows: the focal length of the optical system is 50 mm, the wavelength of the light source is 613 nm, the pupil size is 6.2 mm, the object distance is 71 mm, the depth of field of the system is 0.13 mm, the optimized depth of field range is 2.0 mm, and the phase mask profile is initialized using the three-dimensional phase mask profile. The changes in the profile during the optimization process are shown in the figure. Figures 4a to 4d As shown. The point spread function corresponding to the final surface shape at different distances is as follows Figures 5a to 5c shown.

[0097] The obtained point spread function is convolved with the training set image. The final simulated image pixel size is 2.4 microns, and the sensor pixel size used is 4.8 microns. That is, two times downsampling is performed to make the two sizes match, and Gaussian noise with a variance of 0.01 is added to obtain the image collected on the simulated sensor, as shown in Figures 6a to 6c As shown in the figure, it can be seen that the image consistency is good within the large depth of field range. These images are then input into the neural network for restoration, and finally a clear image with a large depth of field is obtained.

[0098] The test was conducted at the most remote focus position of 1 mm after optimization. Compared with the traditional wavefront coding design under the same conditions, it can be clearly seen that the embodiment of the present invention improves the imaging quality within the extended depth of field while improving the depth of field of the microscopic imaging system. Using the Peak Signal-to-Noise Ratio (PSNR) as a quantitative evaluation indicator, the embodiment of the present invention improved the imaging PSNR of the system by 5.64dB. In summary, the depth of field extension method for underwater in-situ microscopic imagers based on deep learning proposed in the embodiment of the present invention has great practical value.

[0099] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that several equivalent substitutions or obvious variations can be made without departing from the scope of the present invention, and that any equivalent performance or application should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for extending the depth of field of an underwater in-situ microscopic imaging instrument based on deep learning, characterized in that: The following steps are involved: S1. Modeling the optical system to be extended in depth of field as a differentiable mathematical model; S2. Adding a phase mask template using a discretized surface expression into the differentiable mathematical model; S3. Obtain the point spread function of the imaging system from a mathematical model with a phase mask added; Convolving the point spread function with the image and simulating sensor sampling to obtain an image captured by the sensor; S4, inputting the image collected by the sensor into the neural network for image restoration; S5. Taking the imaging quality within the final large depth of field range as the optimization goal, optimize the parameters of the phase mask template and the neural network; Step S1 specifically includes: S11. According to Fourier optics, the point spread function of the image plane (x2, y2) is discretized and modeled as the square of the Fourier transform of the generalized pupil function: PSF(x2,y2)=|F{P(x1,y1)}| 2 Where F(·) represents the two-dimensional Fourier transform process, and P(x1,y1) is the generalized pupil function, which can be expressed as: Among them, (x1, y1) represents each position of the pupil plane, A(x1, y1) represents the amplitude of each position of the pupil plane, φ(x1, y1) represents the phase of each position of the pupil plane, and e i Represents a complex exponential with a natural base; S12. Model the phase change caused by defocus as: in represents a number, λ represents the wavelength of light, z and z0 represent the defocus distance and focus distance of the imaging system respectively, Represents the square of the distance from the coordinate point (x1, y1) on the pupil plane to the center of the pupil plane; Step S2 specifically includes: adding a phase mask to the pupil plane, that is, using an additional phase term φ M (x1,y1) to modulate the generalized pupil function: Where h(x1,y1) is the thickness of the phase mask at each position, and n is the refractive index of the phase mask; The phase term in the modulated generalized pupil function is: φ(x1,y1,z)=φ OOF (x1,y1,z)+φ M (x1,y1).

2. The method for extending the depth of field of an underwater in-situ microscopic imaging instrument based on deep learning according to claim 1, wherein: Step S3 specifically includes: S31. Substitute the phase term in the modulated generalized pupil function into the point spread function calculation formula to calculate the PSF (x2, y2, z) of the optical system with the phase mask added at each position z0, z1, ... within the depth of field range; S32. Use the training image I0(x,y) to convolve with the PSF(x2,y2,z) at different positions of the depth of field to be expanded obtained in the differentiable mathematical model, that is, simulate the imaging image I(x2,y2,z) at different depth of field positions collected by the sensor plane: Among them, η~N(0,σ 2 ) represents zero-mean Gaussian noise with a standard deviation of σ, which represents the noise when the sensor receives the signal. Represents convolution calculation; S33. Calculate the pixel size s in the image I(x2, y2, z) based on the optical system parameters pixel for: Where f is the focal length of the imaging system and d is the diameter of the pupil plane; S34 . Sampling is performed according to the image pixel size and the sensor pixel size to obtain an image S(I) that matches the sensor pixel size.

3. The method for extending the depth of field of an underwater in-situ microscopic imaging instrument based on deep learning according to claim 1, wherein: In step S4, the image restoration includes the following steps: S41. Each decoder layer in the neural network performs two convolutions with a kernel size of 3*3 on the image, and then performs a pooling layer operation on the obtained feature map; S42, after four downsamplings, enters the decoder part, and the image also undergoes two convolutions with a convolution kernel size of 3*3, and then upsamples. At the same time, the final feature map obtained by the decoder of the previous layer is connected with the upsampling result; S43. After completing the four upsamplings, the image undergoes two more convolutions with a convolution kernel size of 3*3, and then undergoes a convolution with a convolution kernel size of 1*1 and is added to the input image to obtain the final output image.

4. The method for extending the depth of field of an underwater in-situ microscopic imaging instrument based on deep learning according to claim 1, wherein: In step S4, the neural network is a ResUnet neural network.

5. The method for extending the depth of field of an underwater in-situ microscopic imaging instrument based on deep learning according to claim 1, wherein: Step S5 specifically includes: S51. The root mean square error between the final output image after image restoration and the training image I0(x,y) is used as the loss function of the system: Among them, m and n represent the number of pixels of the length and width of the image; S52. Optimize the surface function h(x1, y1) of the phase mask and the parameters in the neural network by using the gradient descent method.

6. The method for extending the depth of field of an underwater in-situ microscopic imaging instrument based on deep learning according to claim 5, characterized in that: Step S52 specifically includes: cyclically updating the surface parameters of the phase mask and the parameters of the neural network by the gradient descent method until the number of cycles reaches a set threshold, and then stopping the updating.

7. A system for extending the depth of field of an underwater in-situ microscopic imager based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images