Digital holographic depth-of-field extension imaging method and system based on wavefront coding

Abstract

The invention discloses a digital holographic depth-of-field extended imaging method and system based on wavefront coding, and the method comprises the steps: adjusting the parameters of all parts in a digital holographic microscopic system according to preset cubic phase parameters, and obtaining a candidate digital holographic microscopic system corresponding to the preset cubic phase parameters; adopting a candidate digital holographic microscopic system to image a preset category of target to obtain a first image; processing the first image by adopting a trained deep neural network corresponding to the candidate digital holographic microscopy system to obtain a second image; and acquiring second images corresponding to all candidate digital holographic microscopic systems, comparing all the second images, selecting the second image of which the image quality is greater than a threshold value, and taking the candidate digital holographic microscopic system corresponding to the second image as a final digital holographic microscopic system so as to image a preset category of target. According to the method, the depth-of-field extension optimization speed can be remarkably improved, and the storage space and computing resource consumption are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical fields of optical design and image processing, and particularly relates to a digital holographic depth-of-field extension imaging method and system based on wavefront coding. Background Art

[0002] Optical microscopes are widely used in fields such as clinical diagnosis, biomedicine, and chip detection. Usually, by increasing the numerical aperture (NA) of the objective lens to reduce the point spread function (PSF), the detail resolution is improved, but problems such as increased aberration, rising cost, and reduced depth of field will occur.

[0003] In response to the constraints of numerical aperture and depth of field, two types of solutions have been proposed in the prior art for depth-of-field extension: one is the multi-focus image fusion method. For example, Pieper et al. fuse image slices with different depths of focus. Although the principle is simple, there are problems such as time-consuming scanning and poor applicability to moving samples (such as living biological cells); the other is the wavefront coding method. For example, Dowski et al. modulate the PSF by inserting a phase mask plate on the pupil plane to achieve fast imaging, and subsequent digital image processing is relied on to decode and restore the image. Traditional image restoration methods (inverse filtering, Wiener filtering, Lucy-Richardsion filtering, blind deconvolution, etc.) respectively face problems such as ill-posed problems, limited noise adaptability, time-consuming iteration, dependence on prior knowledge, or complex algorithms, and it is difficult to achieve a balance in terms of stability, reconstruction quality, and restoration efficiency.

[0004] In recent years, deep learning technology has made remarkable progress in the field of image restoration. In 2014, Chao Dong et al. first applied deep learning to image restoration problems. In 2018, Vincent et al. proposed to jointly optimize optical parameters and reconstruction algorithm parameters, realizing the joint optimization of elements such as the diffraction light propagation, depth of field, and image post-processing of a typical camera lens system by an "end-to-end" deep neural network. Since then, the new end-to-end deep learning framework has further enhanced the imaging effect of wavefront coding technology. In 2023, Wang Junhua et al. from Fudan University proposed an end-to-end wavefront coding depth-of-field extension imaging method, which uses wavefront coding technology to jointly optimize the optical design and digital image processing process, extend the depth of field and improve the image quality, achieving the purpose of simplifying the optical system design and reducing the complexity of digital processing algorithms.

[0005] In actual research, it is often necessary to measure cells at large depths or micro-nano components with a large steepness. However, the numerical aperture of traditional microscope objectives limits the simultaneous consideration of the field of view, depth of field, and resolution. As a new type of coherent imaging technology, digital holographic microscopy plays a very important role in the field of three-dimensional measurement with its advantages of non-contact, high sensitivity, and high resolution, and can quickly restore the three-dimensional morphology of an object through a single-frame image. During the operation of flow cytometry using a digital holographic microscopy system, the numerical focusing technology needs to sequentially calculate the reconstructed images at different distances during the restoration process, which is time-consuming and cannot achieve real-time three-dimensional imaging. At the same time, the aforementioned "end-to-end" training method makes less use of the relevant knowledge in the field of image deblurring, has limited restoration ability, and requires huge storage space and computing resources, resulting in a reduced computing speed and an extended computing time. Summary of the Invention

[0006] To solve the above problems existing in the prior art, the present invention provides a digital holographic depth-of-field extended imaging method and system based on wavefront coding. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0007] In a first aspect, the present invention provides a digital holographic depth-of-field extended imaging method based on wavefront coding, including:

[0008] According to the preset cubic phase parameter, adjust the parameters of each component in the digital holographic microscopy system so that the amplitude transfer function of the adjusted digital holographic microscopy system satisfies the consistency of the modulated amplitude transfer function within the corresponding depth of field, and obtain the candidate digital holographic microscopy system corresponding to the preset cubic phase parameter;

[0009] According to multiple preset cubic phase parameters, obtain multiple candidate digital holographic microscopy systems;

[0010] Use the candidate digital holographic microscopy system to image a preset category of targets to obtain a first image; use the trained deep neural network corresponding to the candidate digital holographic microscopy system to process the first image to obtain a second image; obtain the second images corresponding to all candidate digital holographic microscopy systems, compare all the second images, select the second image with an image quality greater than the threshold, and use the candidate digital holographic microscopy system corresponding to the second image as the final digital holographic microscopy system to image the preset category of targets.

[0011] In a second aspect, the present invention further provides a digital holographic depth-of-field extended imaging system based on wavefront coding, including: a digital holographic microscopy system, a first optimization module, and a second optimization module; wherein, the digital holographic microscopy system includes an adjustable attenuator, a polarizer, a beam expander, an objective lens, a first lens, a beam splitter, a second lens, a spatial light modulator, and an image sensor;

[0012] A digital holographic microscopy system is used to receive the laser emitted by a light source. The laser emitted by the light source is processed by an adjustable attenuator and a polarizer to obtain a specific polarized light. The specific polarized light is irradiated onto a sample through a beam expander. After the formed image is magnified by an objective lens, it is divided into a first polarized light and a second polarized light by a first lens and a beam splitter. The second polarized light is irradiated onto a spatial light modulator through a second lens, and wavefront coding is obtained according to the modulation effect. The modulated light is reflected to the beam splitter to interfere with the first polarized light, and an interference fringe image is acquired by an image sensor.

[0013] A first optimization module is configured to adjust the parameters of each component in the digital holographic microscopy system according to a preset cubic phase parameter, so that the amplitude transfer function of the adjusted digital holographic microscopy system satisfies the consistency of the modulation amplitude transfer function within the corresponding depth of field range, and a candidate digital holographic microscopy system corresponding to the preset cubic phase parameter is obtained; multiple candidate digital holographic microscopy systems are obtained according to multiple preset cubic phase parameters.

[0014] A second optimization module is configured to use a candidate digital holographic microscopy system to image a preset category of targets to obtain a first image; use a trained deep neural network corresponding to the candidate digital holographic microscopy system to process the first image to obtain a second image; acquire the second images corresponding to all candidate digital holographic microscopy systems, compare all the second images, select the second image with an image quality greater than a threshold, and use the candidate digital holographic microscopy system corresponding to the second image as the final digital holographic microscopy system to image the preset category of targets.

[0015] Advantages of the present invention:

[0016] A method and system for digital holographic depth-of-field extension imaging based on wavefront coding provided by the present invention combines deep learning and wavefront coding depth-of-field extension technology into the optimization process of the digital holographic microscopy system through progressive optimization. On the premise of significantly improving the depth-of-field extension optimization speed, reducing the storage space and computational resource consumption, the digital holographic microscopy system after depth-of-field extension is used to achieve fast imaging of a single flow cytometry image.

[0017] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Description of the Drawings

[0018] Figure 1 is a flowchart of a method for digital holographic depth-of-field extension imaging based on wavefront coding provided by an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of a digital holographic depth-of-field extension imaging system based on wavefront coding provided by an embodiment of the present invention;

[0020] Figure 3It is a schematic diagram of the deep learning network provided by an embodiment of the present invention;

[0021] Figure 4 It is a schematic diagram of the application of wavefront coding technology provided by an embodiment of the present invention. Specific embodiments

[0022] The present invention will be further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0023] The existing end-to-end wavefront coding depth of field extension technology mainly combines wavefront coding technology by jointly optimizing the optical design layer and the image processing layer, and trains and optimizes the weights of the optical layer and network parameters together, so as to relax the design constraints on the optical system, reduce the complexity of the optical system design, and extend the depth of field of the system to achieve global optimization of optical design and digital processing. However, this technology has the following significant disadvantages:

[0024] 1) High computational resource requirements: Most deep neural networks for million-pixel-level microscopic image restoration require large storage space and computational resources, and it is difficult to be actually deployed and applied in an integrated system.

[0025] 2) Low computational efficiency: Due to the large amount of computation, the computational speed of the existing technology is slow, and it is difficult to meet the requirements of real-time or high-efficiency processing.

[0026] 3) Large storage space occupation: The complex structure and large parameter scale of the deep neural network result in too high storage space occupation, which limits its practical application in resource-constrained integrated systems.

[0027] At the same time, for digital holographic microscopy systems, the existing depth of field extension technology solutions are imperfect and unclear. In the process of flow cytometry imaging, when using digital holographic microscopy numerical focusing technology, since the position of cells in the sample flow is constantly changing, and cells themselves have different three-dimensional structures, morphologies, and depth positions and other characteristics, in order to accurately obtain complete information of cells, including the three-dimensional contour, internal structure details, etc. of cells, it is necessary to take pictures from different focal planes, which greatly reduces the cell detection efficiency.

[0028] In view of this, the present invention provides a digital holographic depth of field extension imaging method and system based on wavefront coding, aiming at:

[0029] 1) Implement a complete solution for the depth of field extension of digital holographic microscopy systems; break the problem that the existing technology can only achieve deep learning-assisted focusing and restoration in the automatic focusing and image restoration processes of digital holographic microscopy systems and cannot perform depth of field extension on the imaging system, and propose a complete depth of field extension solution for digital holographic microscopy systems.

[0030] 1) Reduce the computational load; significantly reduce the computational load of the neural network and improve the computational efficiency by optimizing the algorithm and network structure design.

[0031] 2) Reduce the storage space occupancy; optimize the network parameters and storage structure, reduce the storage space requirements, and make it more suitable for deployment in resource-constrained integrated systems.

[0032] 3) Improve the computational speed; shorten the computational time by improving the computational process and algorithm efficiency to meet the requirements of real-time or high-efficiency processing.

[0033] 4) Promote the technology application; while realizing the depth-of-field extension based on wavefront coding, reduce the complexity and cost of technology implementation, and promote the practical application of wavefront coding depth-of-field extension technology in a wider range of fields.

[0034] By applying deep learning and wavefront coding depth-of-field extension technology to the optimization process of the digital holographic microscopy system through progressive optimization, on the premise of significantly improving the depth-of-field extension optimization speed, reducing the storage space and computational resource consumption, the goal is to use the digital holographic microscopy system after depth-of-field extension to achieve fast imaging of single flow cytometry images.

[0035] First, the present invention needs to build a digital holographic microscopy system. The digital holographic microscopy system includes a light source 1, an adjustable attenuator 2, a polarizer 3, a beam expander 4, a sample 5, an objective lens 6, a first lens 7, a beam splitter 8, a second lens 9, a spatial light modulator 10, and an image sensor 11;

[0036] The digital holographic microscopy system is used to receive the laser emitted by the light source. The laser emitted by the light source is processed by the adjustable attenuator and the polarizer to obtain a specific polarized light. The specific polarized light is irradiated onto the sample through the beam expander. After the formed image is magnified by the objective lens, it is divided into a first polarized light and a second polarized light through the first lens and the beam splitter. The second polarized light is irradiated onto the spatial light modulator through the second lens, and wavefront coding is obtained according to the modulation effect. The modulated light is reflected to the beam splitter to interfere with the first polarized light, and the interference fringe image is obtained through the image sensor. By introducing the spatial light modulator, a wavefront coding element can be used to adjust the parameters of the system according to the phase modulation requirements.

[0037] Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of a method for digital holographic depth-of-field extension imaging based on wavefront coding provided by an embodiment of the present invention, Figure 2 is a schematic diagram of a digital holographic depth-of-field extension imaging system based on wavefront coding provided by an embodiment of the present invention. A method for digital holographic depth-of-field extension imaging based on wavefront coding provided by the present invention includes:

[0038] S101. Adjust the parameters of each component in the digital holographic microscopy system according to the preset cubic phase parameters, so that the amplitude transfer function of the adjusted digital holographic microscopy system satisfies the consistency of the modulated amplitude transfer function within the corresponding depth of field, and obtain a candidate digital holographic microscopy system corresponding to the preset cubic phase parameters.

[0039] Specifically, in this embodiment, according to the requirements of phase modulation, given the preset cubic phase parameters, finely adjust the surface parameters of the lens system to achieve the consistency of the modulation function MTF within the target depth of field, and obtain a suitable candidate digital holographic microscopy system. It can be understood that the parameters of the candidate digital holographic microscopy system include the finely adjusted lens element parameters and the cubic phase parameters corresponding to the spatial light modulator.

[0040] In this embodiment, to obtain candidate models under different phase modulation parameters, different cubic phase amplitudes need to be selected and loaded onto the liquid crystal screen of the spatial light modulator to achieve wavefront coding. At the same time, for different cubic phase amplitudes, it is also necessary to finely adjust the surface parameters of the optical lens of the digital holographic microscopy system, so as to achieve the purpose of consistent amplitude transfer function MTF within the target depth of field interval and obtain candidate models.

[0041] In this embodiment, by adding a cubic phase plate on the pupil plane of the traditional optical system, a wavefront coding system can be obtained, and an intermediate image insensitive to defocus can be obtained. Then, through digital image processing and filtering and restoring the intermediate image, a clear image with a large depth of field can be obtained. After the system adds the phase plate, its normalized pupil coordinates are:

[0042]

[0043] where x represents the spatial coordinate of the aperture stop;

[0044] The modulus of the optical transfer function of the adjusted digital holographic microscopy system is the amplitude transfer function of the adjusted digital holographic microscopy system, and the expression of the optical transfer function of the adjusted digital holographic microscopy system is:

[0045]

[0046] where u represents the normalized spatial frequency, u = 2x / D, D represents the diameter of the pupil, represents the defocus aberration, and α represents the modulation parameter of the cubic phase plate, which controls the degree of phase deviation, that is, the degree of insensitivity to defocus.

[0047] It should be noted that if α is large enough, the second term of the exponent of the optical transfer function OTF can be ignored, and there is no defocus parameter in the formula at this time. It can be considered that the OTF of an optical system with a cubic phase plate has defocus invariance, and its amplitude part, i.e., the modulation transfer function MTF, also has defocus invariance, thus expanding the depth of field of the optical system.

[0048] S102. Obtain a plurality of candidate digital holographic microscopy systems according to a plurality of preset cubic phase parameters.

[0049] S103. Use a candidate digital holographic microscopy system to image a preset category of targets to obtain a first image; use a trained deep neural network corresponding to the candidate digital holographic microscopy system to process the first image to obtain a second image; obtain the second images corresponding to all candidate digital holographic microscopy systems, compare all the second images, select the second image whose image quality is greater than a threshold, and use the candidate digital holographic microscopy system corresponding to the second image as the final digital holographic microscopy system to image the preset category of targets.

[0050] Optionally, according to the clarity of the second image, select the second image whose image quality is greater than a threshold. The first image can be a blurred image, and the second image can be a clear image.

[0051] Specifically, in this embodiment, for each candidate digital holographic microscopy system, obtain a trained deep neural network corresponding to it;

[0052] The process of obtaining the trained deep neural network includes:

[0053] According to the parameters of each component in the candidate digital holographic microscopy system, obtain a plurality of training pairs of preset categories and construct a training data set;

[0054] Input a part of the samples in the training data set into the j-th deep neural network to be trained for training, calculate the loss, and use it as the loss of the j-th training process.

[0055] Perform backpropagation according to the loss of the j-th training process to update the network parameters of the j-th deep neural network to be trained, and obtain the (j + 1)-th deep neural network to be trained; iterate in this way until the number of training times or the convergence degree meets the preset conditions, and obtain the trained deep neural network.

[0056] In this embodiment, according to the parameters of each component in the candidate digital holographic microscopy system, obtaining a plurality of training pairs of preset categories and constructing a training data set includes:

[0057] Construct a candidate digital holographic microscopy system model according to the preset software to obtain a point spread function with a preset depth; optionally, use zemax software;

[0058] Using a microscope to scan, the focal stack images corresponding to the plant tissue sections are obtained; optionally, the sections are tilted at a certain angle, and the area with an axial distance of 100 μm before and after centered on the focal plane is scanned at a step size of 10 μm to obtain 21 focal stack images at different positions. At this time, from the 1st to the 21st, the clear positions in the pictures change linearly;

[0059] Extract the clear imaging areas of each section in the focal stack images, sum the multiple clear image areas to obtain a full-focus image, which is used as Image One;

[0060] Perform a convolution operation on each section in the focal stack images with the point spread function to obtain multiple convolution results; sum all the convolution results to reconstruct the acquisition image of the digital holographic microscopy system, which is used as Image Two; optionally, Image One is a full-focus image, that is, a clear image, and Image Two is an analog encoded image, that is, a blurred image;

[0061] Use Image One and Image Two as a training pair to construct a training dataset; among them, Image One is the true label of the samples in the training dataset.

[0062] In this embodiment, part of the samples in the training dataset are input into the j-th deep neural network to be trained for training, and the loss is calculated and used as the loss in the j-th training process, including:

[0063] In the first training stage, fix the parameters of the generator of the j-th neural network to be trained, train the discriminator of the neural network to be trained, input the Image Two of the training pair in the training dataset into the discriminator, calculate the first loss, input the image generated by the generator into the discriminator, calculate the second loss, and perform backpropagation according to the first loss and the second loss to update the network parameters of the discriminator of the j-th deep neural network to be trained;

[0064] In the second training stage, fix the parameters of the discriminator of the j-th neural network to be trained, train the generator of the neural network to be trained, input the Image One of the training pair in the training dataset into the generator, calculate the adversarial loss and the L1 loss, and perform backpropagation according to the adversarial loss and the L1 loss to update the network parameters of the generator of the j-th deep neural network to be trained.

[0065] Optionally, the generator processes each encoded image to generate a restored image, and then the discriminator evaluates the differences between these generated images and the real images. The whole process adjusts the parameters of the generator and the discriminator through backpropagation to gradually optimize the network performance. The training process of the generative adversarial network is carried out alternately. The generator tries to generate realistic images, while the discriminator tries to distinguish between true and false. During training, the generator and the discriminator are trained separately, but the whole process is end-to-end because the input is directly the original data and the output is the final result.

[0066] 1. The composition of the training pairs includes

[0067] Input group: Different reconstructed encoded images (blurred images);

[0068] Label group: All - focused ground - truth images (clear images);

[0069] 2. The processing flow of the generator includes

[0070] Input: A single encoded blurred image (size 512×512×3);

[0071] Processing: Feature extraction and reconstruction are performed through a U - Net encoder, residual blocks, and a U - Net decoder;

[0072] Output: The restored clear image (same size as the input);

[0073] 3. The processing flow of the discriminator includes

[0074] Input: All - focused ground - truth images or generated images (randomly and alternately input);

[0075] Processing: Extract local image patch features through a PatchGAN architecture;

[0076] Output: A probability map of 70×70×1 (true - false discrimination results for each patch)

[0077] 4. Alternating training:

[0078] Fix the generator and train the discriminator:

[0079] Input: All - focused ground - truth images (label group) → Discriminated as true;

[0080] Input: Generated images (output of the generator) → Discriminated as false;

[0081] Optimize the discriminator parameters through cross - entropy loss;

[0082] Fix the discriminator and train the generator:

[0083] Input: Encoded images → Generate restored images;

[0084] Loss function:

[0085] GAN loss: The generator hopes that the discriminator misjudges as true;

[0086] Optimize the generator parameters through backpropagation.

[0087] In this embodiment, the objective function G of the generator target is;

[0088]

[0089] I L1 (G)=E x,y,z [y - G(x, z)];

[0090] where, min G represents minimizing the objective function with the generator as the optimization object. The purpose is to adjust the generator parameters so that the image G(x, z) output by the generator approximates the all - focused image y, i.e., the second image, as closely as possible, reducing the probability of being recognized as "fake" by the discriminator. max D represents minimizing the objective function with the discriminator as the optimization object. The purpose is to adjust the discriminator parameters so that it can distinguish the all - focused image (x, y) and the generated image (x, G(x, z)) as accurately as possible, strengthening the discrimination ability. D represents the discriminator, G represents the generator, and E x,y represents the mathematical expectation calculated based on the joint distribution of the true label and the second image. That is, for all samples of "label x + all - focused image y", the statistical average value of logD(x, y) is calculated to measure the discrimination effect of the discriminator on the real image. E x,z represents the mathematical expectation calculated based on the joint distribution of the local true label and the image generated by the generator. For the samples of "label x + the input image z of the generator", the statistical average value of 1 - logD(x, G(x, z)) is calculated to reflect the discrimination effect of the discriminator on the generated image. x represents the true label, y represents the second image, z represents the image generated by the generator, and λ represents the hyperparameter used to balance the weights of the adversarial loss and the L1 regularization loss in the objective function. By adjusting λ, the trade - off between the adversarial authenticity and the pixel - level accuracy (the L1 difference from the real image) of the generated image is controlled. I L1 L(G) represents the L1 loss of the generator.

[0091] In this embodiment, the generator is a U - Net model, including an encoder, a residual block, a decoder, a convolutional layer, and an activation function; the first image is processed by a trained deep neural network corresponding to the candidate digital holographic microscopy system to obtain the second image, including:

[0092] The trained encoder is used to perform downsampling on the first image to obtain the first feature;

[0093] The trained residual block is used to process the first feature to obtain the enhanced feature;

[0094] The trained decoder is used to perform upsampling on the enhanced feature to obtain the second feature;

[0095] The trained convolutional layer and activation function are used to process the second feature to obtain the second image.

[0096] In this embodiment, the discriminator is a PatchGAN model.

[0097] Optionally, the generator adopts a pix2pix image conversion model based on the supervised method, and its architecture is a U-net network. Since the data uses paired images, the objective function of pix2pix contains two parts: one is the adversarial loss function, and the other is the L1 regularization term (that is, the modulus of the difference between the generated image and the Ground-truth).

[0098] The discriminator is a PatchGAN model, and the output is a matrix. Each element in the matrix represents a local area of the input image. If the local area is real, 1 is obtained, otherwise 0.

[0099] The original GAN design outputs an evaluation value (True or False) to evaluate the entire image generated by the generator. PatchGAN is in the form of a fully convolutional network. After the image passes through various convolutional layers, it will not be input into the fully connected layer or activation function, but uses convolution to map the input into an N*N matrix, which is equivalent to the final evaluation value in the original GAN to evaluate the generated image of the generator. Each point (true or false) in the N*N matrix represents the evaluation value of a small area in the original image, realizing detail enhancement.

[0100] Please refer to Figure 3 , Figure 3 which is a schematic diagram of a deep learning network provided by an embodiment of the present invention. A deep learning neural network is trained for each group of candidate models respectively. The specific training process is as follows: taking natural plant tissue sections as an example, training pairs are generated through a simulated supervision strategy; a commercial microscope combined with a piezoelectric objective scanner is used to obtain a three-dimensional sample focal stack. Each slice of the captured focal stack is first convolved with a point spread function PSF at a specific depth, and then summed, and the capture is recombined from the integrated microscope. The network architecture is as shown in Figure 3 The generator is mainly based on the U-Net model and consists of a U-Net encoder and decoder module. The U-Net encoder module uses four encoder blocks, and each block consists of a 4×4 convolutional layer (stride = 2); the U-Net decoder module uses four symmetric decoder blocks, and each block consists of bilinear interpolation and a 3×3 convolutional layer; 9 residual blocks are used after the encoder module to further enhance the feature transformation ability of the network; each residual block consists of two 3×3 convolutional layers; the discriminator adopts a standard PatchGAN model, which contains 70 receptive fields from pix2pix. This discriminator architecture can penalize the structure at the scale of local patches to encourage high-frequency details.

[0101] In this embodiment, the generator includes a first convolutional layer, a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, a fifth convolutional module, a sixth convolutional module, a seventh convolutional module, an eighth convolutional module, a ninth convolutional module, and a second convolutional layer. The first convolutional layer is a 3×3 convolutional layer for feature extraction, which extracts local features by convolving and scanning the input data. The second convolutional layer is a 1×1 convolutional layer that optimizes the final output feature form to adapt to the target format of image generation. The first convolutional module, the second convolutional module, and the third convolutional module each include a third convolutional layer and a first activation function. The third convolutional layer is a 4×4 convolutional layer with a stride of 2 for downsampling, reducing the data volume and extracting abstract features. The first activation function is the LeakyReLU activation function to avoid gradient vanishing. The fourth convolutional module and the fifth convolutional module each include a fourth convolutional layer and a second activation function. The fourth convolutional layer is a 3×3 convolutional layer, and the second activation function is the ReLU activation function. The fourth convolutional layer is followed by the second activation function for feature extraction and non-linear transformation to extract feature representation ability. The sixth convolutional module, the seventh convolutional module, the eighth convolutional module, and the ninth convolutional module each include an upsampling module, a fifth convolutional layer, and a third activation function. The fifth convolutional layer is a 3×3 convolutional layer for further processing the features. The third activation function is the ReLU activation function to increase non-linearity. A candidate digital holographic microscopy system is used to image a preset category target to obtain a first image. A trained deep neural network corresponding to the candidate digital holographic microscopy system is used to process the first image to obtain a second image, including:

[0102] Use the trained first convolutional layer to extract features from the first image to obtain local features;

[0103] Use the trained first convolutional module to process the local features to obtain first deep features;

[0104] Use the trained second convolutional module to process the first deep features to obtain second deep features;

[0105] Use the trained third convolutional module to process the second deep features to obtain third deep features;

[0106] Use the trained fourth convolutional module to process the third deep features to obtain first enhanced features;

[0107] Use the trained fifth convolutional module to process the first enhanced features to obtain second enhanced features;

[0108] Use the trained sixth convolutional module to process the second enhanced features to obtain first enhanced non-linear features;

[0109] Process the first enhanced non - linear feature using the trained seventh convolution module to obtain the second enhanced non - linear feature;

[0110] Process the second enhanced non - linear feature using the trained eighth convolution module to obtain the third enhanced non - linear feature;

[0111] Process the third enhanced non - linear feature using the trained ninth convolution module to obtain the fourth enhanced non - linear feature;

[0112] Process the fourth enhanced non - linear feature using the trained second convolution layer to obtain the second image.

[0113] The discriminator includes a tenth convolution module, an eleventh convolution module, a twelfth convolution module, a thirteenth convolution module, a fourteenth convolution module and a fourth activation function. The fourth activation function is the Sigmoid activation function, which is used to output the discrimination result. The tenth convolution module, the eleventh convolution module and the twelfth convolution module all include a sixth convolution layer and a fifth activation function. The sixth convolution layer is a 4×4 convolution layer with a stride of 2, which is used for downsampling operation to reduce the data volume and extract abstract features. The fifth activation function is the LeakyReLU activation function, which is used to avoid gradient disappearance. The thirteenth convolution module and the fourteenth convolution module both include a seventh convolution layer and a sixth activation function. The seventh convolution layer is a 4×4 convolution layer, which is used to extract the local structure features of the image. The sixth activation function is the LeakyReLU activation function, which is used to avoid gradient disappearance.

[0114] In summary, a digital holographic depth - of - field extension imaging method based on wavefront coding provided by the present invention has the following beneficial effects:

[0115] First, the method proposed by the present invention realizes the depth - of - field extension of the digital holographic microscopy system for the first time. By combining the depth - of - field extension scheme based on progressive intelligent optimization of wavefront coding with the digital holographic microscopy system, the optical optimization of the entire digital holographic microscopy system is realized, and the depth - of - field extension function is achieved, solving the problem that the existing flow cytometry detection numerical focusing cannot perform fast imaging.

[0116] Second, a step - by - step progressive joint optimization method; different from the end - to - end optimization method, the present invention proposes the concept of progressive joint optimization. The first - step optimization constrains the range of the best imaging candidate model through the consistency of the MTF. The second - step optimization selects the model by jointly using the deep learning network based on the candidate model obtained in the first step.

[0117] Thirdly, the method proposed in the present invention can significantly reduce memory consumption and computational costs, and can improve the overall optimization speed; by optimizing the algorithm and network structure design, the computational amount of the neural network is significantly reduced, and the computational efficiency is improved; by optimizing network parameters and storage structure, the storage space requirement is reduced, making it more suitable for deployment in resource-constrained integrated systems; by improving the computational process and algorithm efficiency, the computational time is shortened to meet the requirements of real-time or high-efficiency processing; while realizing depth of field extension based on wavefront coding, the complexity and cost of technical implementation are reduced, promoting the practical application of wavefront coding depth of field extension technology in a wider range of fields.

[0118] The present invention first proposes to extend the depth of field of a digital holographic microscopy optical system. The prior art can only achieve autofocus and reconstruction of the images captured by the digital holographic microscopy system through a deep neural network, and cannot achieve the joint optimization of the optical system. Facing the requirement of large-depth-of-field imaging of single cells during flow cytometry detection, fast imaging cannot be achieved, and only multiple images can be taken through autofocus and then image fusion, which greatly slows down the detection speed and increases the subsequent computational amount. The embodiment of the present invention proposes a fast optimization method for depth of field extension of a digital holographic microscopy system based on wavefront coding and an intelligent network, which first realizes the joint optimization of the optical system and imaging, obtains the best performance of depth of field extension, effectively avoids local optimization, realizes high-efficiency joint optimization, and the scalability of this scheme is high. Wavefront coding through a spatial light modulator can also greatly reduce consumables and production costs. In addition, the progressive optimization method proposed in the present invention can effectively reduce the memory occupied by optimization and improve the optimization speed. Different from the end-to-end optimization scheme, this scheme mainly highlights the progressive optimization design. First, a digital holographic microscopy imaging system is constructed, and the wavefront coding function is realized by using a spatial light modulator to replace the three-phase mask template, achieving the first step of depth of field extension optimization. During the first optimization process, the spatial light modulator, as a wavefront coding element, can give a fixed cubic phase parameter according to the phase modulation requirement, and achieve the consistency of the modulation function MTF within the target depth of field range by finely adjusting the surface parameters of the lens system, and finally obtain a suitable candidate model, greatly reducing the range of models that need to be optimized using deep learning subsequently. During the second optimization process, a deep neural network is used to restore the clear image, and a discriminator is established to judge the restoration effect, and then the parameters of the optical element and the modulation coefficient loaded by the spatial light modulator are readjusted, and finally the optimal depth of field extension system optimization design of the digital holographic microscopy system is obtained. The whole set of solutions first proposes a complete solution for depth of field extension of a digital holographic microscopy system, and at the same time effectively reduces the memory occupied by optimization and improves the optimization speed compared with the end-to-end solution.

[0119] Based on the same inventive concept, please continue to refer to Figure 2, the present invention also provides a digital holographic depth-of-field extended imaging system based on wavefront coding, which is used to implement the digital holographic depth-of-field extended imaging method provided in the above embodiments of the present invention. For the embodiments of the method, please refer to the above, and will not be elaborated here; the system includes:

[0120] A digital holographic microscopy system, a first optimization module, and a second optimization module; wherein, the digital holographic microscopy system includes an adjustable attenuator, a polarizer, a beam expander, an objective lens, a first lens, a beam splitter, a second lens, a spatial light modulator, and an image sensor;

[0121] The digital holographic microscopy system is used to receive the laser emitted by the light source. The laser emitted by the light source is processed by the adjustable attenuator and the polarizer to obtain a specific polarized light. The specific polarized light is irradiated onto the sample through the beam expander. After the formed image is magnified by the objective lens, it is divided into a first polarized light and a second polarized light through the first lens and the beam splitter. The second polarized light is irradiated onto the spatial light modulator through the second lens. According to the modulation effect, wavefront coding is obtained. The modulated light is reflected to the beam splitter to interfere with the first polarized light, and the interference fringe image is obtained through the image sensor;

[0122] The first optimization module is used to adjust the parameters of each component in the digital holographic microscopy system according to the preset cubic phase parameters, so that the amplitude transfer function of the adjusted digital holographic microscopy system satisfies the consistency of the modulated amplitude transfer function within the corresponding depth of field, and a candidate digital holographic microscopy system corresponding to the preset cubic phase parameters is obtained; according to multiple preset cubic phase parameters, multiple candidate digital holographic microscopy systems are obtained;

[0123] The second optimization module is used to image a preset category of targets using the candidate digital holographic microscopy system to obtain a first image; process the first image using a trained deep neural network corresponding to the candidate digital holographic microscopy system to obtain a second image; obtain the second images corresponding to all candidate digital holographic microscopy systems, compare all the second images, select the second image with an image quality greater than the threshold, and use the candidate digital holographic microscopy system corresponding to the second image as the final digital holographic microscopy system to image the preset category of targets.

[0124] In an optional embodiment of the present invention, a cubic phase plate is added to the pupil plane of the digital holographic microscopy system, and the function of the cubic phase plate is loaded onto the spatial light modulator.

[0125] In this embodiment, the phase function expression of the cubic phase plate is: φ(x,y) = α(x 3 +y 3) where α represents a coefficient, and the function parameter α is determined through optical design to meet the phase modulation requirements for depth of field extension; the phase value of each pixel is calculated according to the phase function using MATLAB, and the phase value (such as in the range of 0 to 2π) is mapped to the gray value recognizable by the SLM to generate the corresponding phase modulation map; the spatial light modulator is connected to the pupil plane position of the optical system, and the generated phase modulation map is imported.

[0126] Please refer to Figure 4 , Figure 4 FIG. is a schematic diagram of the application of the wavefront coding technology provided by an embodiment of the present invention. An embodiment of the present invention provides a method for rapidly optimizing the depth of field extension of a digital holographic microscopy system based on wavefront coding and an intelligent network. The core principle of this solution is to achieve the depth of field extension of the digital holographic microscopy system, improve the calculation efficiency of the depth of field extension, reduce memory occupancy, and increase the calculation speed through two joint optimizations at different angles. This method provides a complete idea for depth of field extension and imaging. Therefore, according to the core idea of the proposed invention, the digital holographic microscopy imaging device in the proposed embodiment can be replaced by other integrated complex microscopy imaging optical paths to achieve small-volume, wide-angle, large-depth-of-field, and high-precision microscopy imaging, and be applied to environments such as component flaw detection, skin pathology observation, and field portable microscopy. As Figure 4 shown, the light source 1 emits laser light, and specific polarized light is obtained through the adjustable attenuator 2 and the polarizer 3. After passing through the beam expander 4, it irradiates the sample 5. The image formed is magnified by the objective lens 6, and then passes through the lens 7 and the beam splitter 8 and is divided into two polarized light beams. One beam passes through the lens 9 and hits the liquid crystal surface of the spatial light modulator 10, and wavefront coding is obtained according to the modulation effect. The modulated light passes through the beam splitter 8 and interferes with the original reference light beam to generate interference fringes at the CCD image sensor 11 and is finally recorded. The optical path shown in the figure can be replaced by various types of complex imaging optical paths that require depth of field extension, such as an endoscope module. By placing a phase plate or a spatial light modulator at the pupil position and using the same optimization idea, the depth of field extension optimization of the endoscope can be achieved and applied to research on aspects such as endoscope disease detection. At the same time, this solution can also be extended to telescopes and surveillance systems to achieve multi-scenario applications such as component flaw detection, skin pathology examination, and field portable microscopy.

[0127] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant are intended to cover non-exclusive inclusion, so that an article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the article or device comprising said element. Similar words such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The orientation or positional relationship indicated by "upper", "lower", "left", "right", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation on the present invention.

[0128] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0129] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A digital holographic depth of field extended imaging method based on wavefront coding, characterized in that: include: According to the preset cubic phase parameter, the parameters of each component in the digital holographic microscopy system are adjusted so that the amplitude transfer function of the adjusted digital holographic microscopy system satisfies the consistency of the modulation amplitude transfer function within the corresponding depth of field, and a candidate digital holographic microscopy system corresponding to the preset cubic phase parameter is obtained; According to a plurality of preset cubic phase parameters, a plurality of candidate digital holographic microscopy systems are obtained; Using the candidate digital holographic microscope system to image a preset category target to obtain a first image; The first image is processed by using a trained deep neural network corresponding to the candidate digital holographic microscopy system to obtain a second image; the second images corresponding to all the candidate digital holographic microscopy systems are obtained, all the second images are compared, the second image with an image quality greater than a threshold is selected, and the candidate digital holographic microscopy system corresponding to the second image is used as the final digital holographic microscopy system to image the preset category target.

2. The digital holographic depth of field extended imaging method based on wavefront coding according to claim 1, characterized in that: The modulus of the adjusted optical transfer function of the digital holographic microscopy system is the adjusted amplitude transfer function of the digital holographic microscopy system. The expression of the adjusted optical transfer function of the digital holographic microscopy system is: Where u represents the normalized spatial frequency, u = 2x / D, D represents the diameter of the pupil, represents the defocus aberration, and α represents the modulation parameter of the cubic phase plate.

3. The digital holographic depth of field extended imaging method based on wavefront coding according to claim 1, characterized in that: Also includes: For each of the candidate digital holographic microscopy systems, obtaining a corresponding trained deep neural network; The process of acquiring the trained deep neural network includes: According to the parameters of each component in the candidate digital holographic microscopy system, a plurality of training pairs of preset categories are obtained to construct a training data set; Input some samples in the training data set into the j-th deep neural network to be trained for training, calculate the loss, and use it as the loss of the j-th training process; Back propagation is performed based on the loss of the j-th training process to update the network parameters of the j-th deep neural network to be trained, and the j+1-th deep neural network to be trained is obtained; this is repeated until the number of training times or the degree of convergence meets the preset conditions, and a trained deep neural network is obtained.

4. The digital holographic depth of field extended imaging method based on wavefront coding according to claim 3 is characterized in that: The step of acquiring a plurality of training pairs of preset categories according to the parameters of each component in the candidate digital holographic microscopy system and constructing a training data set comprises: Constructing the candidate digital holographic microscopy system model according to preset software to obtain a point spread function at a preset depth; Microscope scanning is used to obtain the focal pile image corresponding to the plant tissue section; Extracting a clear imaging area of ​​each slice in the focus pile image, summing a plurality of clear image areas, and obtaining a fully focused image as image one; Performing a convolution operation on each slice in the focus pile image and the point spread function to obtain a plurality of convolution results; summing all the convolution results to reconstruct an image collected by the digital holographic microscope system as image 2; The image one and the image two are used as a training pair to construct the training data set; wherein the image one is the true label of the sample in the training data set.

5. The digital holographic depth of field extended imaging method based on wavefront coding according to claim 3, characterized in that: The step of inputting some samples in the training data set into the j-th deep neural network to be trained for training, calculating the loss, and using the loss as the j-th training process loss includes: In the first training stage, the parameters of the generator of the j-th neural network to be trained are fixed, the discriminator of the neural network to be trained is trained, the image 2 of the training pair in the training data set is input to the discriminator, the first loss is calculated, the image generated by the generator is input to the discriminator, the second loss is calculated, and back propagation is performed according to the first loss and the second loss to update the network parameters of the discriminator of the j-th deep neural network to be trained; In the second training stage, the parameters of the discriminator of the j-th neural network to be trained are fixed, the generator of the neural network to be trained is trained, the image of the training pair in the training data set is input into the generator, the adversarial loss and the L1 loss are calculated, and back propagation is performed according to the adversarial loss and the L1 loss to update the network parameters of the generator of the j-th deep neural network to be trained.

6. The digital holographic depth of field extended imaging method based on wavefront coding according to claim 5, characterized in that: The objective function G of the generator target for; I L1 (G)=E x,y,z [y-G(x,z)]; Among them, min G It means that the generator is optimized to find the minimum value of the objective function, max D The discriminator is used as the optimization object to find the minimum value of the objective function. D represents the discriminator, G represents the generator, and E represents the x,y represents the mathematical expectation calculated based on the joint distribution of the true label and the second image, E x,z represents the mathematical expectation of the joint distribution calculation of the local true label and the image generated by the generator, x represents the true label, y represents image 2, z represents the image generated by the generator, λ represents the hyperparameter, I L1 (G) represents the L1 loss of the generator.

7. The digital holographic depth of field extended imaging method based on wavefront coding according to claim 5, characterized in that: The generator is a U-Net model, including an encoder, a residual block, a decoder, a convolutional layer and an activation function; the first image is processed using a trained deep neural network corresponding to the candidate digital holographic microscopy system to obtain a second image, including: Using the trained encoder to downsample the first image to obtain a first feature; Using the trained residual block to process the first feature to obtain an enhanced feature; Using a trained decoder to perform an upsampling operation on the enhanced feature to obtain a second feature; The second feature is processed using the trained convolutional layer and the activation function to obtain a second image.

8. The digital holographic depth of field extended imaging method based on wavefront coding according to claim 5, characterized in that: The discriminator is a patchGAN model.

9. A digital holographic depth-of-field extended imaging system based on wavefront coding, characterized in that: include: A digital holographic microscopy system, a first optimization module and a second optimization module; wherein the digital holographic microscopy system comprises an adjustable attenuator, a polarizer, a beam expander, an objective lens, a first lens, a beam splitter, a second lens, a spatial light modulator and an image sensor; The digital holographic microscope system is used to receive laser light emitted by a light source. The laser light emitted by the light source is processed by the adjustable attenuator and the polarizer to obtain specific polarized light. The specific polarized light is irradiated onto the sample through the beam expander. The formed image is magnified by the objective lens and then separated into a first beam of polarized light and a second beam of polarized light through the first lens and the beam splitter. The second beam of polarized light is irradiated onto the spatial light modulator through the second lens. Wavefront coding is obtained according to the modulation effect. The modulated light is reflected to the beam splitter to generate interference with the first beam of polarized light, and an interference fringe image is obtained through the image sensor. The first optimization module is used to adjust the parameters of each component in the digital holographic microscopy system according to the preset cubic phase parameter, so that the amplitude transfer function of the adjusted digital holographic microscopy system satisfies the consistency of the modulation amplitude transfer function within the corresponding depth of field, and obtain a candidate digital holographic microscopy system corresponding to the preset cubic phase parameter; according to multiple preset cubic phase parameters, multiple candidate digital holographic microscopy systems are obtained; The second optimization module is used to use the candidate digital holographic microscopy system to image a preset category target to obtain a first image; use a trained deep neural network corresponding to the candidate digital holographic microscopy system to process the first image to obtain a second image; obtain the second images corresponding to all the candidate digital holographic microscopy systems, compare all the second images, select the second image with an image quality greater than a threshold, and use the candidate digital holographic microscopy system corresponding to the second image as the final digital holographic microscopy system to image the preset category target.

10. The digital holographic extended depth of field imaging system based on wavefront coding according to claim 9, characterized in that: A cubic phase plate is added to the pupil plane of the digital holographic microscopy system, and the function of the cubic phase plate is loaded into the spatial light modulator.

Images