Digital holographic microscopic distortion compensation method and system combined with deep learning

Abstract

The invention relates to a digital holographic microscopic distortion compensation method and system combined with deep learning. According to the method, a deep learning image segmentation technology and a phase distortion compensation technology are fused, an off-axis transmission-type digital holographic microscopic system is built, a hologram is shot to serve as a data set, then a U-net convolutional neural network is improved, and the improved U-net convolutional neural network is trained by using the data set; and segmenting the tested sample and the background in the hologram, and fitting the unwrapped background phase by using the least square based on the Zernike polynomial so as to realize phase distortion compensation. The invention further provides a digital holographic microscopic distortion compensation system combined with deep learning. The system comprises a module training module, an image processing module and a phase compensation module. Compared with the prior art, the method has the advantages of effectively eliminating distortion generated by the digital holographic system, improving efficiency, improving accuracy, saving calculation cost, improving network training effect and generalization ability and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of digital holography, and in particular to a digital holographic microscopic distortion compensation method combined with deep learning and a system thereof. Background Art

[0002] In digital holographic microscopy technology, there is a problem of phase distortion. Aberration refers to the imaging defect caused by non-ideal optical properties in the actual optical system. In digital holograms, aberration is mainly manifested as the deviation between the actual imaging and the ideal imaging. Distortion is a type of aberration, which will change the shape of the object. Due to the influence of various factors, such as imperfections in the optical system and environmental interference, the collected digital holographic images have phase distortion, which affects the image quality and subsequent analysis and processing.

[0003] The traditional double exposure method has high requirements on system stability. In actual experiments, it is difficult to ensure that there is no new disturbance in the system between two exposures. It is difficult to deal with complex phase distortion and the compensation effect is not ideal. The principal component analysis method requires that aberrations dominate, which may not be met in actual conditions. The applicable conditions are limited and preprocessing is required to increase the complexity.

[0004] In the actual data, there may be some outliers in the ordinary least squares method, which will have a great impact on the regression results. Because the least squares method determines the parameters by minimizing the sum of squares of errors, the presence of outliers will increase the sum of squares, causing the regression coefficient to shift in the direction of the outliers, making the fitting results inaccurate. When the amount of data is large or the dimension is high, these operations may be very time-consuming or even computationally infeasible, especially in the modern big data era, the data scale is constantly increasing, and the computational efficiency of the ordinary least squares method is low.

[0005] The invention patent with publication number CN115760598A discloses a method for compensating digital holographic wrapped phase distortion based on deep learning. The random Zernike polynomial coefficients and the corresponding wrapped phase map are generated by computer and used as learning labels and network training neural network models respectively; the hologram is numerically reconstructed and its wrapped phase map is input into the trained neural network model to output the Zernike polynomial coefficients to reconstruct the phase distortion distribution and compensate for the residual distortion, but there are problems of complex processing and low efficiency.

[0006] The invention patent with publication number CN110309910A discloses an optimized adaptive microscopic imaging method and device based on machine learning, which uses an ultrashort pulse laser to collect image data through a point scanning method; constructs a convolutional neural network to train the network with simulation results obtained by inputting image data into a physical model; applies the trained network to an adaptive method to optimize the imaging results and eliminate image distortion, and uses a model fitting method to obtain the optimal phase compensation for system and sample distortion correction, but there are problems of inaccurate fitting results and low efficiency. Summary of the invention

[0007] The purpose of the present invention is to provide a digital holographic microscopic distortion compensation method and system combined with deep learning in order to overcome the defects of the above-mentioned prior art. By combining the improved deep learning image segmentation technology with the traditional least squares fitting algorithm, accurate compensation for phase distortion can be achieved, solving the problems of limited accuracy, poor adaptability, high computational complexity and low reliability in the prior art.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] According to one aspect of the present invention, a digital holographic microscopy distortion compensation method combined with deep learning is provided. The method integrates deep learning image segmentation technology with phase distortion compensation technology, firstly segments the sample under test and the background in the hologram, and then performs distortion compensation; the method steps include:

[0010] S1. Build an off-axis transmission digital holographic microscopy system, and use the system to take multiple holograms of the sample under test containing different background information as a data set;

[0011] S2, train the improved U-net convolutional neural network using the data set in S1; obtain the trained U-net convolutional neural network;

[0012] S3, inputting the single hologram to be processed into the trained U-net convolutional neural network to obtain a phase mask;

[0013] S4, unwrapping the single hologram to be processed to obtain an unwrapped phase;

[0014] S5, combining the phase mask and the unwrapped phase to obtain the unwrapped background phase of the single hologram to be processed;

[0015] S6. Fit the unwrapped background phase using the least square method based on Zernike polynomials to achieve phase distortion compensation.

[0016] As a preferred technical solution, the off-axis transmission digital holographic microscopy system in S1 is constructed using a laser with a wavelength of 671nm.

[0017] As a preferred technical solution, the off-axis transmission digital holographic microscope system in S1 is constructed by components fixed on a resolution board, including a laser, a half-wave plate, a beam expander collimator, a beam splitter, a microscope, a lens, and a CCD.

[0018] As a preferred technical solution, the specific steps of taking multiple shots of the holograms of the sample under test containing different background information as a data set in S1 are:

[0019] S11. Shoot and adjust the vehicle fine-tuning knob of the resolution board;

[0020] S12, fixing the step length, controlling the resolution plate to move in different directions, and obtaining holograms in different directions, positions and distances;

[0021] S13, perform data enhancement on the hologram to obtain a data set. As a preferred technical solution, in S2, the number of U-net neural network layers is deepened, and the initial channel is set to 8.

[0022] As a preferred technical solution, the improved U-net neural network in S2 uses group normalization instead of batch normalization.

[0023] As a preferred technical solution, a mixed loss function is used in S2 when training the improved U-net convolutional neural network. The mixed loss function includes a binary cross entropy loss function, an intersection-over-union loss function, and a multi-scale structural similarity loss function, and its expression is:

[0024]

[0025] Among them: k is a mixed loss function; is the binary cross entropy loss function; is the intersection-over-union loss function; is the multi-scale structural similarity loss function; is a hyperparameter of each loss function.

[0026] As a preferred technical solution, in S6, the Zernike polynomial is used to perform least square fitting on the unwrapped phase diagram, so as to obtain the total phase distortion of the system, which is expressed as:

[0027]

[0028] in, is the total phase distortion of the system; w i (x,y) is the weight; a iare the coefficients of each term of the Zernike polynomial.

[0029] As a preferred technical solution, in the phase distortion compensation implemented in S6, the phase value of the measured sample after distortion compensation is the unwrapped phase minus the total phase distortion of the system, and the expression is:

[0030]

[0031] in, is the phase value of the measured sample after distortion compensation; is the unwrapping phase; is the total phase distortion of the system;.

[0032] According to another aspect of the present invention, a digital holographic microscopy distortion compensation system combined with deep learning is provided, characterized in that the system applies a digital holographic microscopy distortion compensation method combined with deep learning as described above to work, and the system includes an off-axis transmission digital holographic microscopy system, a module training module, an image processing module and a phase compensation module;

[0033] An off-axis transmission digital holographic microscope system is used to take multiple holograms of the sample under test containing different background information as a data set;

[0034] Model training module, used to train convolutional neural networks using datasets;

[0035] An image processing module is used to solve the phase mask and unwrapped phase of a single hologram to be processed, and then obtain an unwrapped phase map;

[0036] The phase compensation module is used to fit the unwrapped background phase based on the least squares of Zernike polynomials to achieve phase distortion compensation.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] 1. The present invention builds an off-axis transmission digital holographic microscope system, and uses the system to repeatedly shoot holograms of a sample under test containing different background information as a data set; uses the data set to train an improved U-net convolutional neural network; obtains a trained U-net convolutional neural network; inputs a single hologram to be processed into the trained U-net convolutional neural network to obtain a phase mask; unpacks the single hologram to be processed to obtain an unpacked phase; combines the phase mask and the unpacked phase to obtain the unpacked background phase of the single hologram to be processed; and then uses the least squares based on Zernike polynomials to fit the unpacked background phase to achieve phase distortion compensation.

[0039] , to achieve phase distortion compensation. Effectively eliminate most of the distortion generated by the digital holographic system, improve efficiency, and achieve precise compensation of digital holographic microscopy automatic phase aberration without human intervention.

[0040] 2. In the construction of the digital holographic optical path of the present invention, the off-axis transmission digital holographic microscope system is constructed using a laser with a wavelength of 671nm. The laser with a wavelength of 671nm emits red laser, which has relatively strong penetrating ability. In some sample observations that require penetration depth, the red laser has a longer wavelength and better penetration ability, and can detect the information inside the sample more deeply and obtain more complete object light information. It is more advantageous for studying the internal structure and three-dimensional morphology of the sample, making the subsequent phase distortion compensation more accurate.

[0041] 3. In the construction of the data set, the present invention performs data enhancement on the hologram, completes the expansion of the data set, and enhances the robustness of the network; in the improved U-net neural network, a padding operation with a value of 1 is performed on the feature map before convolution to ensure that the input and output of the network have the same size; thereby expanding the scope of application of the method.

[0042] 4. The present invention improves the U-net neural network, deepens the number of U-net neural network layers, and sets the initial channel to 8, thereby improving computing efficiency and saving computing costs.

[0043] 5. The improved U-net neural network of the present invention uses group normalization instead of batch normalization to reduce the impact of too small batch size on network accuracy, thereby improving the training effect and generalization ability of the network.

[0044] 6. In the present invention, a hybrid loss function is used when training the improved U-net convolutional neural network. The hybrid loss function includes a binary cross entropy loss function, an intersection-over-union loss function, and a multi-scale structural similarity loss function, so as to narrow the gap between the segmented image and the label image and obtain a clear boundary of the high-confidence segmented image. The binary cross entropy loss function can maintain the smooth gradient of all pixels. The intersection-over-union loss function emphasizes regional similarity, directly optimizes the overlap between the predicted and true segmented regions, and handles the problem of imbalance in the number of foreground and background pixels, which helps to improve the accuracy of the overall segmented region. The multi-scale structural similarity loss function calculates image similarity at multiple scales through multi-scale analysis, comprehensively considers details at different resolutions, optimizes the global and local structures of the image, and helps to improve the overall quality of the segmentation and detail retention. The combination of the three loss functions provides a more comprehensive and robust image quality assessment, applies a larger loss near the boundary to optimize the segmentation details, thereby improving the accuracy of image segmentation and making subsequent phase distortion compensation more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of the digital holographic microscopy distortion compensation method combined with deep learning in the present invention;

[0046] Figure 2 This is a schematic diagram of the improved U-net network architecture in the present invention;

[0047] Figure 3a This is a schematic diagram of the U-net output result before improvement in the embodiment;

[0048] Figure 3b This is a schematic diagram of the improved U-net output result in the embodiment;

[0049] Figure 4a This is a diagram showing the effect of distortion compensation before improvement in the embodiment;

[0050] Figure 4b This is a diagram showing the improved distortion compensation effect in the embodiment;

[0051] Figure 5a This is a schematic diagram of the U-net output result before improvement in the embodiment;

[0052] Figure 5b This is a schematic diagram of the improved U-net output result in the embodiment;

[0053] Figure 6a This is a diagram showing the effect of distortion compensation before improvement in the embodiment;

[0054] Figure 6b This is a diagram showing the improved distortion compensation effect in the embodiment;

[0055] Figure 7 Schematic diagram of the off-axis transmission digital holographic microscopy system constructed in the embodiment. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] Example

[0058] In this embodiment, a digital holographic microscopy distortion compensation method combined with deep learning is applied. The method integrates deep learning image segmentation technology with phase distortion compensation technology, firstly segments the sample under test and the background in the hologram, and then performs distortion compensation. The method steps are as follows: Figure 1 As shown, including:

[0059] S1. Build an off-axis transmission digital holographic microscope system, and use the system to take multiple holograms of the sample under test containing different background information as a data set;

[0060] In this embodiment, the off-axis transmission digital holographic microscopy system is constructed by components fixed on a resolution board, such as Figure 7 As shown, the components include laser, half-wave plate, beam expander collimator, beam splitter, microscope, lens, and CCD.

[0061] In this embodiment, the fourth group of digital elements of the USAF1951 calibration plate hologram are first photographed, and by adjusting the carrier fine-tuning knob of the fixed resolution plate, the movement of the calibration plate in different directions is precisely controlled under the premise of a fixed step size to obtain holograms in different directions, positions and distances.

[0062] In this embodiment, the USAF1951 resolution plate is used. First, a 671nm red laser emits a high-coherence laser, and the intensity ratio of the object light to the reference light is adjusted by a λ / 2 half-wave plate; the laser is converted from a spherical wave to a plane wave by a beam expander collimator, and the laser spot is uniform; it is divided into two lights of equal intensity by a BS1 beam splitter, one of which penetrates the object and carries the object information through the microscope MO1 and lens L1 to form the object light path; the other beam, after adjusting the intensity ratio of the object light to the reference light by a λ / 2 half-wave plate, is reflected by a reflector M, and directly passes through the microscope MO2 and lens L2 to form the reference light path. The object light is refracted by BS3, and finally interferes with the reference light at BS2, and the information obtained is collected by the CCD.

[0063] In this embodiment, each image of the data set is rotated clockwise by 90°, 180° and 270° respectively. The above method is used to enhance the data set, complete the expansion of the data set and enhance the robustness of the network, and finally a data set with 1,000 holograms is obtained for the training of the improved U-net convolutional neural network.

[0064] S2, train the improved U-net convolutional neural network using the data set in S1; obtain the trained U-net convolutional neural network;

[0065] In this embodiment, in order to make the traditional U-net neural network more suitable for digital holography, it is improved. The improved U-net neural network architecture is as follows: Figure 2 shown.

[0066] In this embodiment, the improved U-net neural network performs a padding operation with a value of 1 on the feature map before convolution, so as to ensure that the input and output of the network have the same size. The improved U-net neural network deepens the number of network layers to further improve the learning ability of the neural network. Considering the efficiency, the initial channel is set to 8 to maintain the calculation efficiency and save the calculation cost.

[0067] In this embodiment, group normalization (GN) is used instead of the commonly used batch normalization (BN) to reduce the impact of too small batch size on network accuracy and thus improve the training effect and generalization ability of the network. For the input x, its dimension is (N, C, H, W), which represents a batch containing N samples, each of which has C channels, and the image size of each channel is H (height) and W (width). The number of channels C is divided into G groups, each with C / G channels. The calculation formula of GN is:

[0068]

[0069] Where i is a region in the image; j is a specific position in this region; μ ig is the mean; is the variance; ε is a constant used to avoid division by zero errors; γ and β are trainable parameters.

[0070] In this embodiment, the improved network uses a hybrid loss function to narrow the gap between the segmented image and the label image, and obtain a clear boundary of the high-confidence segmented image. The hybrid loss function expression is:

[0071]

[0072] in: is the binary cross entropy (BCE) loss function, is the Intersection over Union (IOU) loss function, is the multi-scale structural similarity index (MS-SSIM) loss function; is a hyperparameter of each loss function.

[0073] In this embodiment, the BCE loss function is commonly used for binary classification and segmentation tasks and is defined as:

[0074]

[0075] Among them, y i is the true label of the i-th pixel; is the predicted probability of the i-th pixel; N is the total number of pixels in the image.

[0076] In this embodiment, the IOU loss function directly optimizes the overlap between the predicted and true segmented regions, handles the problem of imbalance in the number of foreground and background pixels, and helps improve the accuracy of the overall segmented region. The expression is as follows:

[0077] IOU=∣A∪B∣∣A∩B∣;

[0078] Among them, A is the predicted segmentation area; B is the real segmentation area; |A∩B| is the number of pixels of the intersection of the predicted area and the real area; |A∪B| is the number of pixels of the union of the predicted area and the real area.

[0079] In this embodiment, the MS-SSIM loss function calculates image similarity at multiple scales, comprehensively considers details at different resolutions, optimizes the global and local structures of the image, and helps improve the overall quality of segmentation and detail retention. The calculation formula of MS-SSIM can be divided into several parts:

[0080] Similarity of brightness, contrast and structure: For the image of the jth scale, the calculation formula of SSIM is as follows:

[0081]

[0082] Among them, μ x,j and μ y,j is the local average of image x and image y at the jth scale; and is the local variance of image x and image y at the jth scale; σ x,jy,j is the local covariance of image x and image y at the jth scale; C1 and C2 are constants to avoid division by zero errors.

[0083] Multi-scale weighted average: The SSIM values ​​of each scale are weighted averaged to obtain the final MS-SSIM value. The formula is as follows:

[0084]

[0085] Where M is the number of scales; ω j is the weight of the jth scale; it is used to adjust the contribution of different scales.

[0086] In this embodiment, when the three loss functions are combined, the BCE loss function can maintain the smooth gradient of all pixels, the IOU loss function focuses on regional similarity, and the MS-SSIM loss function provides a more comprehensive and robust image quality evaluation through multi-scale analysis. It optimizes the segmentation details by applying a larger loss near the boundary, thereby improving the accuracy of image segmentation.

[0087] S3, inputting the single hologram to be processed into the trained U-net convolutional neural network to obtain a phase mask;

[0088] S4, unwrapping the single hologram to be processed to obtain an unwrapped phase;

[0089] S5, combining the phase mask and the unwrapped phase to obtain the unwrapped background phase of the single hologram to be processed;

[0090] S6. Fit the unwrapped background phase using the least square method based on Zernike polynomials to achieve phase distortion compensation.

[0091] In this embodiment, through deep learning image segmentation, we can effectively extract the sample area under test from the hologram and obtain the target phase mask. By combining the phase mask with the unwrapped phase containing the sample under test and system distortion, The unwrapped phase diagram excluding the phase value of the measured sample is obtained, and finally the Zernike polynomial is used to perform the least squares fitting on the unwrapped phase diagram to obtain the total phase distortion of the system. The fitting expression is as follows:

[0092]

[0093] in, is the total phase distortion of the system; w i (x,y) is the weight; a i are the coefficients of each term of the Zernike polynomial.

[0094] In this embodiment, the least squares fitting is used to solve the coefficients a i , the solution process is as follows:

[0095]

[0096] Where K represents the number of sampling points in the background area.

[0097] In this embodiment, the fitting coefficient matrix equation obtained by the solution process is as follows:

[0098]

[0099] In this embodiment, the phase value of the measured sample after distortion compensation can be expressed as follows:

[0100]

[0101] in, is the phase value of the measured sample after distortion compensation; is the unwrapping phase; is the total phase distortion of the system;.

[0102] In this embodiment, by using deep learning image segmentation technology, we can accurately segment the object under test and its background in the hologram, thereby eliminating the interference caused by the background during the phase value fitting process. This makes the measurement of the phase value of the sample under test more accurate. Compared with traditional image processing technology, the phase distortion compensation technology combined with deep learning has obvious advantages in both computational efficiency and segmentation accuracy, making the distortion compensation result more accurate.

[0103] In this embodiment, if Figure 3a , Figure 3b , Figure 5a , Figure 5b As shown in the figure, the improved U-net is sharper and more accurate in the segmentation of the edge of the sample under test. It can better capture the edge details, making the segmented sample outline clearer and more consistent with the boundary of the actual sample; the improved network can better retain small-scale features. It can accurately segment the sample under test, while the unimproved network may lose some subtle structural information.

[0104] In this embodiment, as shown in 4a, Figure 4b , Figure 6a , Figure 6b As shown, the improved network accurately identifies the boundary between the sample area and the background area, making the phase distortion compensation more accurate and effectively eliminating the distortion generated by the digital holographic microscopy system, thereby obtaining a more accurate height of the sample under test. Compared with the unimproved method, the method of the present invention has significant advantages in segmentation accuracy, distortion compensation effect, and accuracy of the height measurement of the sample under test.

[0105] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A digital holographic microscopic distortion compensation method combined with deep learning, characterized in that: This method combines deep learning image segmentation technology with phase distortion compensation technology, first segmenting the sample under test and the background in the hologram, and then performing distortion compensation; the method steps include: S1. Build an off-axis transmission digital holographic microscope system, and use the system to take multiple holograms of the sample under test containing different background information as a data set; S2, train the improved U-net convolutional neural network using the data set in S1; obtain the trained U-net convolutional neural network; S3, inputting the single hologram to be processed into the trained U-net convolutional neural network to obtain a phase mask; S4, unwrapping the single hologram to be processed to obtain an unwrapped phase; S5, combining the phase mask and the unwrapped phase to obtain the unwrapped background phase of the single hologram to be processed; S6. Fit the unwrapped background phase using the least square method based on Zernike polynomials to achieve phase distortion compensation.

2. The method for compensating distortion of digital holographic microscopy combined with deep learning according to claim 1, characterized in that: The off-axis transmission digital holographic microscopy system in S1 is constructed by components fixed on a resolution board, including a laser, a half-wave plate, a beam expander collimator, a beam splitter, a microscope, a lens, and a CCD.

3. The method for compensating distortion of digital holographic microscopy combined with deep learning according to claim 2, characterized in that: The specific steps of taking multiple shots of the hologram of the sample under test containing different background information as a data set in S1 are: S11. Shoot and adjust the vehicle fine-tuning knob of the resolution board; S12, fixing the step length, controlling the resolution plate to move in different directions, and obtaining holograms in different directions, positions and distances; S13. Perform data enhancement on the hologram to obtain a data set.

4. The method for compensating distortion of digital holographic microscopy combined with deep learning according to claim 1, characterized in that: The improved U-net neural network in S2 performs a padding operation with a value of 1 on the feature graph before convolution to ensure that the input and output of the network have the same size.

5. The method for compensating distortion of digital holographic microscopy combined with deep learning according to claim 1, characterized in that: In S2, the number of U-net neural network layers is deepened and the initial channel is set to 8.

6. The method for compensating digital holographic microscopic distortion combined with deep learning according to claim 1, characterized in that: The improved U-net neural network in S2 uses group normalization instead of batch normalization.

7. The method for compensating distortion of digital holographic microscopy combined with deep learning according to claim 1, characterized in that: In S2, a mixed loss function is used when training the improved U-net convolutional neural network. The mixed loss function includes a binary cross entropy loss function, an intersection-over-union loss function, and a multi-scale structural similarity loss function, and its expression is: Where: lk is the mixed loss function; is the binary cross entropy loss function; is the intersection-over-union loss function; is the multi-scale structural similarity loss function; is a hyperparameter of each loss function.

8. The method for compensating distortion of digital holographic microscopy combined with deep learning according to claim 1, characterized in that: In S6, the Zernike polynomial is used to perform least square fitting on the unwrapped phase diagram, thereby obtaining the total phase distortion of the system, which is expressed as: in, is the total phase distortion of the system; w i (x,y) is the weight; a i are the coefficients of each term of the Zernike polynomial.

9. The method for compensating digital holographic microscopic distortion combined with deep learning according to claim 1, characterized in that: In the phase distortion compensation implemented in S6, the phase value of the measured sample after distortion compensation is the unwrapped phase minus the total phase distortion of the system, and the expression is: in, is the phase value of the measured sample after distortion compensation; is the unwrapping phase; is the total phase distortion of the system;.

10. A digital holographic microscopic distortion compensation system combined with deep learning, characterized in that: The system applies a digital holographic microscopy distortion compensation method combined with deep learning as described in claims 1-9 to work, and the system includes an off-axis transmission digital holographic microscopy system, a module training module, an image processing module and a phase compensation module; An off-axis transmission digital holographic microscope system is used to take multiple holograms of the sample under test containing different background information as a data set; Model training module, used to train convolutional neural networks using datasets; An image processing module is used to solve the phase mask and unwrapped phase of a single hologram to be processed, and then obtain an unwrapped phase map; The phase compensation module is used to fit the unwrapped background phase based on the least squares of Zernike polynomials to achieve phase distortion compensation.

Images