Phase reconstruction method for differential interference contrast microscopy based on pix2pix network

Abstract

The phase reconstruction method of differential interference contrast microscopy based on pix2pix network includes the following steps: S1, constructing an end-to-end deep learning strategy based on pix2pix network, including a generator and a discriminator, each including an input channel and an output channel; S2, collecting and constructing a data set; S3, training the pix2pix network; S4, improving the ability of the network to generate accurate phase reconstruction images through training; S5, by combining the error curves of the training set and the test set, the network performance can be comprehensively analyzed; S6, the trained network is used for quantitative phase reconstruction of differential interference contrast; S7, the trained network is used for quantitative phase reconstruction of differential interference contrast. This solution can solve the problem of difficulty in achieving high-precision, artifact-free, dynamic phase reconstruction in differential interference contrast technology.

Description

Technical Field

[0001] The invention relates to the technical field of optical differential interference contrast, in particular to a differential interference contrast microscopy phase reconstruction method based on a pix2pix network. Background Art

[0002] Optical differential interference contrast (DIC) microscopy is an optical microscopy technique used to enhance the contrast of transparent samples so that they can show clear structures without staining. Based on the shearing interference of partially coherent light, differential interference contrast technology maps the gradient of the sample's optical path length to the image intensity to indirectly record the physical properties of the sample, making it very sensitive to tiny phase gradients. Due to its high contrast, high resolution and optical sectioning capabilities, optical differential interference contrast microscopy has been widely used in biology, materials science, cytology and other fields, and is particularly suitable for observing transparent samples such as cells, tissue sections, microorganisms, and polymer films. In live cell imaging, DIC is particularly favored by researchers because it can provide high-contrast images without the need for staining.

[0003] According to the geometric optical imaging model, the DIC image is formed by the coherent superposition of two image field copies u(r) and u(r+δr) with a spatial displacement δr, and its intensity can be expressed as:

[0004]

[0005] Among them, |u(r) 2 and The phase shifter is used to introduce a phase delay θ into an image field. n =nπ / 2 (n=0,1,2,3), and keep the other image field unchanged. Based on the four intensity images, the differential phase of the sample along the displacement direction can be solved As shown in Equation 1, the actual phase of the sample to be tested is Differential phase images obtained using DIC microscopy With the following relationship:

[0006]

[0007] Therefore, DIC microscopy can convert tiny refractive index differences in transparent samples into light and dark contrast, significantly enhancing the visibility of the sample and being very sensitive to tiny phase gradients. And due to the common path system and phase shear, the DIC system has better vibration resistance and lower system noise.

[0008] Despite its great success and widespread application, DIC still has many limitations. One of them is that its intensity image has a nonlinear relationship with the optical path length gradient of the sample along the shear direction, which makes it a very challenging task to reconstruct a quantitative phase image directly related to the original physical properties of the sample, such as refractive index, thickness or optical path length, from the DIC image.

[0009] On the one hand, based on the DIC imaging model, a linear integration method was developed for phase reconstruction of DIC images. However, due to the unknown integration constant, noise accumulation, and amplification of low-frequency noise during the integration process, there are serious directional artifacts in the reconstructed image during the line-by-line integration along the shear direction. On the other hand, obtaining the complete boundary information of the sample through multiple DIC images with different shear directions can greatly improve the accuracy and robustness of phase reconstruction. In order to obtain DIC images with different shear directions, it is usually necessary to rotate the sample or shear prism, and the alignment or defocus problems introduced by this operation need to be further corrected. Based on this, how to achieve artifact-free and dynamic phase reconstruction while ensuring high precision is still a difficulty of this technology. Summary of the invention

[0010] In view of the above-mentioned defects, the purpose of the present invention is to propose a phase reconstruction method based on deep learning and differential interference contrast, which can solve the problem of difficulty in achieving high-precision, artifact-free, dynamic phase reconstruction in differential interference contrast technology, and provide an effective method for dynamic phase reconstruction of differential interference contrast.

[0011] To achieve this object, the present invention adopts the following technical solutions:

[0012] The phase reconstruction method of differential interference contrast microscopy based on pix2pix network includes the following steps:

[0013] S1. Build an end-to-end deep learning strategy based on the pix2pix network, which includes a generator and a discriminator, each of which includes an input channel and an output channel;

[0014] S2. According to the phase shift amount of equal step size / fixed step size, a total of 5000 sets of digital holographic phase shift interferograms of different samples of Hela cells and polystyrene microspheres were collected to generate the real phase for network training, and the differential phase was calculated according to the preset shear amount and shear direction; 5000 sets of "differential phase and real phase" data sets were constructed, of which 4000 sets of data were used for network training as training sets, and 1000 sets were used for network testing as test sets; the input was the differential phase and the label was the real phase;

[0015] S3, train the pix2pix network, and set the differential phase obtained in the training set As input, the real phase As the label of the network; during the training process, a combination of adversarial loss and similarity loss is used to reflect the difference between the network output and its corresponding label map. The gradient descent method is used in training to optimize the parameters in the network;

[0016] S4. Improve the network's ability to generate accurate phase reconstruction images through training. The purpose of network training is to make the Loss function converge. If not, change the network parameters or adjust the network architecture. In addition, in order to further study the network accuracy and convergence, input the training set and test set data alternately during the training process, and record the error curves of the training set and test set.

[0017] S5. By combining the error curves of the training set and the test set, the network performance can be comprehensively analyzed; based on the analysis of network accuracy and convergence, the network parameters and loss function can be optimized;

[0018] S6. The trained network is used for quantitative phase reconstruction of differential interference contrast. The DIC images with phase shifts of 0, π / 2, π, and 3π / 2 collected by DIC microscopy are differentially phase-constructed using a four-step phase-shift algorithm. Quantitative recovery of the differential phase Input network, can directly get the sample phase

[0019] S7. Use the trained network for quantitative phase reconstruction in differential interference contrast.

[0020] Preferably, in step S2, an off-axis Mach-Zehnder digital holographic interference optical path is constructed, and the phase shift amounts are 0, π / 2, π, and 3π / 2, respectively, and a digital holographic phase shift interference pattern is collected for the sample.

[0021] Preferably, in step S2, the digital holographic phase shift interferograms with phase shift amounts of 0, π / 2, π, and 3π / 2 are subjected to a four-step phase shift algorithm to obtain a sample phase, and then the sample phase is added to the calculated background phase using a threshold segmentation method to generate a real phase for network training.

[0022] A computer is used to randomly generate a background phase that is consistent with the noise of the DIC optical path, which is used to replace the background phase of the sample reconstructed by the digital holographic interference optical path, so that the data set is closer to the phase distribution of the real DIC image;

[0023] The image is obtained by translating the real phase image using the image sub-pixel displacement method based on Fourier transform And the differential phase is obtained according to the formula

[0024]

[0025] in, is the differential phase, is the real phase, δr is the spatial displacement, is the image after the real phase shift δr;

[0026] The relationship between the translation distance and the microscope shear distance needs to be consistent with the shear distance of the DIC microscope.

[0027] Preferably, in step S3, the specific training goal can be expressed as:

[0028]

[0029] Among them, G is the generator, D is the discriminator, and L cGAN (G,D) and L L2 (G) represents adversarial loss and similarity loss respectively, λ represents the regularization parameter, which is set to 100; the adversarial loss is as follows:

[0030] L cGAN (G,D)=E x,y [logD(x,y)]+E x [1-logD(x,G(x))] (4)

[0031] Where E represents the expected value, x represents the differential phase of the input, y represents the integral label phase, and G(x) represents the output of the generator. It can be seen that the generator and the discriminator compete and play against each other with this function as the target. The similarity loss is as follows:

[0032] L L2 (G) = E x,y [||yG(x)||2] (5)

[0033] Here, the L2 norm is introduced, which is different from the L1 function commonly used in the Pix2Pix architecture because the common integral phase is relatively continuous, which can better constrain the output of the generator to be close to the truth.

[0034] Preferably, in step S5, three situations may occur when analyzing the network performance. First, if the errors of the training set and the test set are both low, it is considered to be moderate fitting and the network parameters are retained; second, if the errors of the training set and the test set are both high, it is considered to be underfitting, and the number of convolutional layers of the network generator is increased and the parameters are adjusted, and the network is retrained; third, if the error of the training set is low and the error of the test set is high, it is considered to be overfitting, and the data set is increased using the above method, and the network is retrained.

[0035] One of the above technical solutions includes the following beneficial effects: constructing an end-to-end deep learning strategy based on a pix2pix network to directly achieve high-precision artifact-free reconstruction of the phase in a differential interference contrast microscope; only a differential phase image sheared in a single direction can achieve artifact-free reconstruction of the sample phase, greatly improving the spatial phase sensitivity of the system and fully displaying the sample details. The method of the present invention uses a simple device, has excellent noise resistance and extremely high robustness; the method does not require multiple iterations, thereby improving the time resolution of phase reconstruction of the differential interference contrast microscope. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the overall steps of an embodiment of the present invention;

[0037] Figure 2 is a schematic diagram of building an end-to-end deep learning strategy based on a pix2pix network according to an embodiment of the present invention;

[0038] Figure 3 It is a specific structure of the discriminator and the generator in the pix2pix network of an embodiment of the present invention;

[0039] Figure 4 The collected phase-shift DIC image (a1), the differential phase image obtained by four-step phase shift calculation (a2), the real phase result directly calculated using the digital holographic optical path (a3), and the reconstructed phase result obtained using this technology (a4) are shown;

[0040] Figure 5 From left to right, the differential phase map (input), the true phase (label), and the phase predicted by the network (output) are shown. DETAILED DESCRIPTION

[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0042] like Figure 1-5 As shown, the differential interference contrast microscopy phase reconstruction method based on the pix2pix network includes the following steps:

[0043] S1. Build an end-to-end deep learning strategy based on the pix2pix network, which includes a generator and a discriminator, each of which includes an input channel and an output channel;

[0044] S2. According to the phase shift amount of equal step size / fixed step size, a total of 5000 sets of digital holographic phase shift interferograms of different samples of Hela cells and polystyrene microspheres were collected to generate the real phase for network training, and the differential phase was calculated according to the preset shear amount and shear direction; 5000 sets of "differential phase and real phase" data sets were constructed, of which 4000 sets of data were used for network training as training sets, and 1000 sets were used for network testing as test sets; the input was the differential phase and the label was the real phase;

[0045] S3, train the pix2pix network, and set the differential phase obtained in the training set As input, the real phase As the label of the network; during the training process, a combination of adversarial loss and similarity loss is used to reflect the difference between the network output and its corresponding label map. The gradient descent method is used in training to optimize the parameters in the network;

[0046] S4. Improve the network's ability to generate accurate phase reconstruction images through training. The purpose of network training is to make the Loss function converge. If not, change the network parameters or adjust the network architecture. In addition, in order to further study the network accuracy and convergence, input the training set and test set data alternately during the training process, and record the error curves of the training set and test set.

[0047] S5. By combining the error curves of the training set and the test set, the network performance can be comprehensively analyzed; based on the analysis of network accuracy and convergence, the network parameters and loss function can be optimized;

[0048] S6. The trained network is used for quantitative phase reconstruction of differential interference contrast. The DIC images with phase shifts of 0, π / 2, π, and 3π / 2 collected by DIC microscopy are differentially phase-constructed using a four-step phase-shift algorithm. Quantitative recovery of the differential phase Input network, can directly get the sample phase

[0049]

[0050] S7. Use the trained network for quantitative phase reconstruction in differential interference contrast.

[0051] An end-to-end deep learning strategy based on the pix2pix network is constructed to directly realize high-precision artifact-free reconstruction of the phase in differential interference contrast microscopy; only the differential phase image sheared in a single direction can realize artifact-free reconstruction of the sample phase, which greatly improves the spatial phase sensitivity of the system and fully displays the sample details. The method of the present invention uses a simple device, has excellent noise resistance and extremely high robustness; the method does not require multiple iterations, thereby improving the phase reconstruction time resolution of differential interference contrast microscopy.

[0052] Among them, in step S2, an off-axis Mach-Zehnder digital holographic interferometer optical path is built, and the phase shift amounts are 0, π / 2, π, and 3π / 2 respectively, and a digital holographic phase shift interferogram is collected for the sample.

[0053] Accurately capture digital holographic phase-shift interferograms.

[0054] In step S2, the digital holographic phase-shift interferograms with phase shifts of 0, π / 2, π, and 3π / 2 are subjected to a four-step phase-shift algorithm to obtain the sample phase, and then the threshold segmentation method is used to add the sample phase to the calculated background phase to generate the real phase for network training.

[0055] A computer is used to randomly generate a background phase that is consistent with the noise of the DIC optical path, which is used to replace the background phase of the sample reconstructed by the digital holographic interference optical path, so that the data set is closer to the phase distribution of the real DIC image;

[0056] The image is obtained by translating the real phase image using the image sub-pixel displacement method based on Fourier transform And the differential phase is obtained according to the formula

[0057]

[0058] in, is the differential phase, is the real phase, δr is the spatial displacement, is the image after the real phase shift δr;

[0059] The relationship between the translation distance and the microscope shear distance needs to be consistent with the shear distance of the DIC microscope.

[0060] The data set is accurately constructed and the noise level is kept the same as the experimental optical path. The shear distance and shear amount are consistent with the experiment, which is more consistent with the actual physical model.

[0061] Among them, in step S3, the specific training goal can be expressed as:

[0062]

[0063] Among them, G is the generator, D is the discriminator, and L cGAN (G,D) and L L2 (G) represents adversarial loss and similarity loss respectively, λ represents the regularization parameter, which is set to 100; the adversarial loss is as follows:

[0064] L cGAN (G,D)=E x,y [logD(x,y)]+E x [1-logD(x,G(x))] (4)

[0065] Where E represents the expected value, x represents the differential phase of the input, y represents the integral label phase, and G(x) represents the output of the generator. It can be seen that the generator and the discriminator compete and play against each other with this function as the target. The similarity loss is as follows:

[0066] L L2 (G) = E x,y [||yG(x)||2] (5)

[0067] Here, the L2 norm is introduced, which is different from the L1 function commonly used in the Pix2Pix architecture because the common integral phase is relatively continuous, which can better constrain the output of the generator to be close to the truth.

[0068] Through the pix2pix network for modal conversion, the generator can better predict the true phase. The discriminator is responsible for judging the validity of the generator output and feeding it back to the network to better achieve integral phase reconstruction.

[0069] In addition, in step S5, three situations may occur when analyzing the network performance. First, if the errors of the training set and the test set are both low, it is considered to be moderate fitting and the network parameters are retained; second, if the errors of the training set and the test set are both high, it is considered to be underfitting, and the number of convolutional layers of the network generator is increased and the parameters are adjusted, and the network is retrained; third, if the error of the training set is low and the error of the test set is high, it is considered to be overfitting, and the data set is increased using the above method, and the network is retrained.

[0070] Prevent the pix2pix network from overfitting and maintain the optimal performance of the network for the task of inverting the integral phase from the differential phase.

[0071] The technical principle of the present invention is described above in conjunction with specific embodiments. These descriptions are only for explaining the principle of the present invention and cannot be interpreted as limiting the scope of protection of the present invention in any way. Based on the explanations herein, those skilled in the art can associate other specific implementations of the present invention without paying creative labor, and these methods will fall within the scope of protection of the present invention.

Claims

1. A phase reconstruction method for differential interference contrast microscopy based on a pix2pix network, characterized in that: The following steps are involved: S1. Build an end-to-end deep learning strategy based on the pix2pix network, which includes a generator and a discriminator, each of which includes an input channel and an output channel; S2. According to the phase shift amount of equal step size / fixed step size, a total of 5000 sets of digital holographic phase shift interferograms of different samples of Hela cells and polystyrene microspheres were collected to generate the real phase for network training, and the differential phase was calculated according to the preset shear amount and shear direction; 5000 sets of "differential phase and real phase" data sets were constructed, of which 4000 sets of data were used for network training as training sets, and 1000 sets were used for network testing as test sets; the input was the differential phase and the label was the real phase; S3, train the pix2pix network, and set the differential phase obtained in the training set As input, the real phase As the label of the network; during the training process, a combination of adversarial loss and similarity loss is used to reflect the difference between the network output and its corresponding label map. The gradient descent method is used in training to optimize the parameters in the network; S4. Improve the network's ability to generate accurate phase reconstruction images through training. The purpose of network training is to make the Loss function converge. If not, change the network parameters or adjust the network architecture. In addition, in order to further study the network accuracy and convergence, input the training set and test set data alternately during the training process, and record the error curves of the training set and test set. S5. By combining the error curves of the training set and the test set, the network performance can be comprehensively analyzed; based on the analysis of network accuracy and convergence, the network parameters and loss function can be optimized; S6. The trained network is used for quantitative phase reconstruction of differential interference contrast. The DIC images with phase shifts of 0, π / 2, π, and 3π / 2 collected by DIC microscopy are differentially phase-constructed using a four-step phase-shift algorithm. Quantitative recovery of the differential phase Input network, can directly get the sample phase S7. Use the trained network for quantitative phase reconstruction in differential interference contrast.

2. The differential interference contrast microscopy phase reconstruction method based on pix2pix network according to claim 1, characterized in that: In step S2, an off-axis Mach-Zehnder digital holographic interferometer optical path is constructed, and the phase shift amounts are 0, π / 2, π, and 3π / 2, respectively, and a digital holographic phase shift interferogram is collected for the sample.

3. The differential interference contrast microscopy phase reconstruction method based on pix2pix network according to claim 2, characterized in that: In step S2, the digital holographic phase shift interferograms with phase shift amounts of 0, π / 2, π, and 3π / 2 are subjected to a four-step phase shift algorithm to obtain the sample phase, and then the threshold segmentation method is used to add the sample phase to the calculated background phase to generate the real phase for network training. A computer is used to randomly generate a background phase that is consistent with the noise of the DIC optical path, which is used to replace the background phase of the sample reconstructed by the digital holographic interference optical path, so that the data set is closer to the phase distribution of the real DIC image; The image is obtained by translating the real phase image using the image sub-pixel displacement method based on Fourier transform And the differential phase is obtained according to the formula in, is the differential phase, is the real phase, δr is the spatial displacement, is the image after the real phase shift δr; The relationship between the translation distance and the microscope shear distance needs to be consistent with the shear distance of the DIC microscope.

4. The differential interference contrast microscopy phase reconstruction method based on pix2pix network according to claim 3, characterized in that: In step S3, the specific training objective can be expressed as: Among them, G is the generator, D is the discriminator, and L cGAN (G,D) and L L2 (G) represents adversarial loss and similarity loss respectively, λ represents the regularization parameter, which is set to 100; the adversarial loss is as follows: L cGAN (G,D)=E x,y [logD(x,y)]+E x [1-logD(x,G(x))](4) Where E represents the expected value, x represents the differential phase of the input, y represents the integral label phase, and G(x) represents the output of the generator. It can be seen that the generator and the discriminator compete and play against each other with this function as the target. The similarity loss is as follows: L L2 (G)=E x,y [||y-G(x)||2](5) Here, the L2 norm is introduced, which is different from the L1 function commonly used in the Pix2Pix architecture because the common integral phase is relatively continuous, which can better constrain the output of the generator to be close to the truth.

5. The differential interference contrast microscopy phase reconstruction method based on pix2pix network according to claim 3, characterized in that: In step S5, three situations may occur when analyzing the network performance. First, if the errors of the training set and the test set are both low, it is considered to be moderate fitting and the network parameters are retained; second, if the errors of the training set and the test set are both high, it is considered to be underfitting, and the number of convolutional layers of the network generator is increased and the parameters are adjusted, and the network is retrained; third, if the error of the training set is low and the error of the test set is high, it is considered to be overfitting, and the data set is increased using the above method, and the network is retrained.

Images