Unsupervised deep learning holographic imaging method and device

Abstract

The present invention proposes an unsupervised deep learning holographic imaging method and device, wherein the method includes collecting a hologram of a target to be trained; obtaining a corresponding holographic sub-image by propagating the hologram in free space, and constructing a data set containing the hologram and the holographic sub-image; constructing a neural network whose input is the holographic sub-image and output is the hologram, and using the data set to train the neural network to obtain a trained neural network. The present invention utilizes the feature data extraction capability of deep learning to learn the potential deep relationship between measurement values, while solving the problems of difficulty in obtaining true values ​​and insufficient imaging fidelity in deep learning, and can achieve large-scale real-time holographic imaging.

Description

Technical Field

[0001] The invention belongs to the field of computational photography. Background Art

[0002] Holography is a traditional method of recording and reconstructing the complete amplitude and phase information of the wavefront. Ordinary cameras can only record the light intensity distribution of light waves emitted or scattered by an object, but cannot record the phase. Holographic technology uses the principle of interference to record the complex amplitude distribution of the object light wave in a certain plane in the form of interference fringes and store them in a recording medium. The recorded interference fringe pattern is called a hologram. When the light wave illuminates the hologram, the original object light wave can be reproduced due to the diffraction effect, forming a three-dimensional image containing all the information of the original object. However, the exposure time required for the recording medium of traditional holography is long, so the stability of the device is required to be high, and it cannot be reproduced in real time.

[0003] With the improvement of parameters of photosensitive electronic imaging devices such as CCD and COMS, and the application of high-speed electronic computers, digital holography has been widely used, such as shape measurement, defect identification, particle size and position measurement. Digital holography uses computers to numerically reconstruct digitized holograms, and can obtain wavefront recovery images at different positions through digital focusing for further quantitative analysis. In the past two decades, some scholars have extracted the phase and amplitude information of the wavefront from three or more interference patterns through the phase shift method, and then used Fresnel transform to reconstruct the target wavefront. However, both the application of time phase shift and the acquisition of interference patterns require relatively long time, which reduces the possibility of digital holography for real-time monitoring.

[0004] Deep learning is a branch of machine learning. It combines low-level features to form more abstract high-level representations of attribute categories or features, thereby extracting different feature information. In recent years, deep learning technology has been introduced into digital holography and has shown great application potential in different fields. The trained end-to-end reconstruction network can solve the problem of real-time monitoring, but the training of deep neural networks requires the real values ​​corresponding to the hologram as a data set, and in practice, the real values ​​are often difficult to obtain. At the same time, deep neural networks trained based on real values ​​can often only learn simple end-to-end mapping relationships from a large number of training sets, and it is difficult to learn the underlying physical model. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the first objective of the present invention is to propose an unsupervised deep learning holographic imaging method, which is used to solve the problems of difficulty in obtaining true values ​​and insufficient imaging fidelity in deep learning.

[0007] The second object of the present invention is to propose an unsupervised deep learning holographic imaging device.

[0008] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes an unsupervised deep learning holographic imaging method, comprising:

[0009] Collecting a hologram of the target to be trained;

[0010] Acquire a corresponding hologram by propagating the hologram in free space, and construct a data set including the hologram and the hologram;

[0011] The input of the construction is the hologram, the output is a neural network of the hologram, the neural network is trained using the data set, and a trained neural network is obtained.

[0012] In addition, the unsupervised deep learning holographic imaging method according to the above embodiment of the present invention may also have the following additional technical features:

[0013] Furthermore, in one embodiment of the present invention, it also includes:

[0014] Collecting a hologram of the target to be restored;

[0015] The hologram is input into the trained neural network, and a reconstructed image of the target to be restored is output.

[0016] Further, in one embodiment of the present invention, the step of acquiring a corresponding holographic sub-image by propagating the hologram in free space includes:

[0017] By means of computer simulation, the hologram is propagated a distance d in free space to obtain a holographic sub-image with a distance sample of 2d, and then a data set including the hologram and the holographic sub-image is constructed.

[0018] Furthermore, in one embodiment of the present invention, the using the data set to train the neural network includes:

[0019] The hologram data set is used as the reference data of the output end of the neural network, and the hologram sub-image data set is input into the network as the input end data for training. The model parameters of the neural network are optimized through a predetermined number of forward and backward propagations.

[0020] Furthermore, in one embodiment of the present invention, the step of inputting the hologram into the trained neural network and outputting a reconstructed image of the target to be restored comprises:

[0021] The image of the target to be restored is reconstructed through the nonlinear characterization capability of the trained neural network and the redundancy of model information between holograms.

[0022] To achieve the above-mentioned purpose, the second aspect of the present invention proposes an unsupervised deep learning holographic imaging method, comprising the following modules:

[0023] An acquisition module, used for acquiring a hologram of a target to be trained;

[0024] A construction module, used for acquiring a corresponding holographic sub-image by propagating the hologram in free space, and constructing a data set including the hologram and the holographic sub-image;

[0025] A training module is used to construct a neural network whose input is the holographic sub-image and whose output is the hologram, and to train the neural network using the data set to obtain a trained neural network.

[0026] Furthermore, in one embodiment of the present invention, it also includes a reconstruction module, which is used to:

[0027] Collecting a hologram of the target to be restored;

[0028] The hologram is input into the trained neural network, and a reconstructed image of the target to be restored is output.

[0029] Furthermore, in one embodiment of the present invention, the building block is also used to:

[0030] By means of computer simulation, the hologram is propagated a distance d in free space to obtain a holographic sub-image with a distance sample of 2d, and then a data set including the hologram and the holographic sub-image is constructed.

[0031] Furthermore, in one embodiment of the present invention, the training module is also used to:

[0032] The hologram data set is used as the reference data of the output end of the neural network, and the hologram sub-image data set is input into the network as the input end data for training. The model parameters of the neural network are optimized through a predetermined number of forward and backward propagations.

[0033] Furthermore, in one embodiment of the present invention, the reconstruction module is further used to:

[0034] The image of the target to be restored is reconstructed through the nonlinear characterization capability of the trained neural network and the redundancy of model information between holograms.

[0035] The unsupervised deep learning holographic imaging method and device proposed in the embodiments of the present invention utilize the feature data extraction capability of deep learning to learn the potential deep relationship between measurement values, while solving the problems of difficulty in obtaining true values ​​and insufficient imaging fidelity in deep learning, and can realize large-scale real-time holographic imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A flowchart of an unsupervised deep learning holographic imaging method provided in an embodiment of the present invention.

[0038] Figure 2 A schematic diagram of a holographic imaging system provided by an embodiment of the present invention.

[0039] Figure 3 A diagram of the computer reconstruction process of unsupervised training provided in an embodiment of the present invention.

[0040] Figure 4 A schematic diagram of the process of an unsupervised deep learning holographic imaging device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0042] The following describes the unsupervised deep learning holographic imaging method and device according to an embodiment of the present invention with reference to the accompanying drawings.

[0043] Example 1

[0044] Figure 1 A flowchart of a holographic imaging method based on unsupervised deep learning provided in an embodiment of the present invention.

[0045] like Figure 1 As shown, the unsupervised deep learning holographic imaging method includes the following steps:

[0046] S101: Collecting a hologram of a target to be trained;

[0047] S102: Acquire a corresponding holographic sub-image by propagating the hologram in free space, and construct a data set including the hologram and the holographic sub-image;

[0048] S103: Construct a neural network whose input is the holographic sub-image and whose output is the hologram, train the neural network using the data set, and obtain the trained neural network.

[0049] Furthermore, in one embodiment of the present invention, it is characterized in that it also includes:

[0050] Collecting a hologram of the target to be restored;

[0051] The hologram is input into the trained neural network, and the reconstructed image of the target to be restored is output.

[0052] Among them, a hologram of the target is obtained. After the modulated laser irradiates the target, a diffraction pattern is formed in the far field. Photosensitive devices such as CCD and CMOS are used as detectors to collect the measurement value of the hologram at a distance d from the sample. Sensors such as CCD and CMOS can only sense light intensity, so the phase information of the wavefront will be lost. The goal of holographic reconstruction is to achieve high-fidelity reconstruction of the original image while suppressing the interference of twin images and zero-order images during the reproduction process.

[0053] Furthermore, in one embodiment of the present invention, it is characterized in that the corresponding holographic sub-image is obtained by propagating the hologram in free space, comprising:

[0054] By means of computer simulation, the hologram is propagated in free space for a distance d to obtain a holographic sub-image with a distance sample of 2d, and then a data set including the hologram and the holographic sub-image is constructed.

[0055] The hologram is propagated in free space to obtain the corresponding holographic sub-image. According to the relationship between the spectrum of the propagation surface and the initial spectrum The propagation process of light in free space is simulated by computer. The collected hologram is propagated for a distance d in free space to obtain a holographic sub-image with a distance sample of 2d. Then a data set containing holograms and holographic sub-images is constructed.

[0056] Furthermore, in one embodiment of the present invention, it is characterized in that using the data set to train the neural network includes:

[0057] The hologram dataset is used as the reference data for the output of the neural network, and the hologram sub-image dataset is used as the input data for input into the network for training. The model parameters of the neural network are optimized through a predetermined number of forward and backward propagations.

[0058] Among them, holograms and holographic sub-images are used as data pairs to train deep neural networks. In the experiment, the data pair D = {Z2, Z1} consisting of holograms and holographic sub-images is selected as the training set of the network model. Using a deep neural network, {Z2} corresponds to the set of holographic sub-images of z = 2d as input, and {Z1} is the set of holograms corresponding to z = d as output reference. After multiple forward and backward propagations, the optimal model parameters are finally achieved.

[0059] Furthermore, in one embodiment of the present invention, it is characterized in that the hologram is input into a trained neural network, and the reconstructed image of the target to be restored is output, including:

[0060] The image of the target to be restored is reconstructed through the nonlinear representation ability of the trained neural network and the redundancy of model information between holograms.

[0061] The hologram is input into the trained neural network. The hologram to be restored is input into the trained network, and the hologram is reconstructed through the nonlinear characterization ability of the neural network and the redundancy of the model information between the holograms. For different disturbances (including noise, distance, etc.), the trained model shows good stability and can achieve real-time and rapid reconstruction of the hologram.

[0062] Example 2

[0063] The experimental device is shown in the figure Figure 2 As shown, the light emitted by the laser source is collimated by the lens to generate parallel light that illuminates the sample. The CCD is used to record the pure amplitude hologram of the missing phase at a distance z=d from the sample. The holographic sub-image at z=2d can be obtained by calculating the free space propagation algorithm of the computer, which can greatly reduce the time for collecting data.

[0064] If the object light intensity is very weak compared to the reference light (generally the ratio of the object light intensity to the reference light intensity is less than 0.1), the expression for the hologram intensity distribution at a distance z from the sample is:

[0065] I(x,y,z)=|1+u(x,y,z)| 2 =1+|u(x,y,z)| 2 +u(x,y,z)+u * (x,y,z),

[0066] Among them, * represents the complex conjugate term, and u(x,y,z) represents the complex amplitude distribution of the object light at a distance z from the sample. The first two terms in the above formula are zero-order derivative images, which can be eliminated by subtracting the average light intensity of the hologram:

[0067] I(x,y,z)=I(x,y,z)-I avg ≈u(x,y,z)+u * (x,y,z),

[0068] Performing Fourier transform on I(x,y,z) yields:

[0069]

[0070] Among them, H(f x ,f y ,z) is the free space propagation function, U(fx ,f y ,0) is the two-dimensional Fourier transform result of u(x,y,0), and the expression is:

[0071]

[0072] It can be seen that the Fourier change L(x,y,z) of the measured value I(x,y,z) is similar to the Fourier change U(f x ,f y ,0) and its conjugate term in the frequency domain. Then there is a nonlinear physical relationship between the target wavefront u(x,y,0) and the hologram I(x,y,z), let z = d, let:

[0073] u(x,y,z)=R{I(x,y,d)},

[0074] Among them, R{·} represents the physical relationship between the target wavefront u(x,y,0) and the hologram measurement value I(x,y,z). Based on this recursive relationship, it is not difficult to infer that the same physical relationship exists between the u(x,y,0) measurement value I(x,y,2d). That is:

[0075] u(x,y,z)=R{I(x,y,2d)},

[0076] Therefore, the neural network learns this physical model through the redundant information between the hologram set {Z2} at z=2d and the hologram set {Z1} at z=d, and approximately infers the mapping relationship between the hologram at z=d and the real wavefront at z=0, and then reconstructs the target image through the hologram.

[0077] More specifically, if Figure 3 As shown in the figure, the hologram collected by the CCD is stored and propagated in free space by a computer to obtain a hologram sub-image set, and a training pair consisting of a hologram and a hologram sub-image is obtained. The hologram set is used as the reference data at the output end of the network, and the hologram sub-image is input into the network as the input end data for training. The nonlinear representation ability of the neural network enables it to learn the potential physical model in the process from z = 2d to z = d; and in the process of hologram reconstruction, only the image of z = d needs to be input to reconstruct the image corresponding to z = 0 through the learned model, that is, the real image. In the process of training, there is no need to obtain the real value, and only the hologram data set needs to be obtained to obtain the hologram sub-image data set through calculation, which reduces the difficulty and time of obtaining the data set.

[0078] A supervised method is used to train the coaxial digital hologram based on the U-net coaxial digital hologram reconstruction neural network. U-net uses the training pair D = {Z2, Z1} as the training data set. {Z2} corresponds to the set of holographic subimages of z = 2d as input, and {Z1} is the set of holograms corresponding to z = d as output reference. In this method, the L1 distance and L2 distance between the perceptual features of the generated image and the perceptual features of the reference image are used to comprehensively judge and measure the difference between the output image and the real image.

[0079] The unsupervised deep learning holographic imaging method and device proposed in the embodiments of the present invention utilize the feature data extraction capability of deep learning to learn the potential deep relationship between measurement values, while solving the problems of difficulty in obtaining true values ​​and insufficient imaging fidelity in deep learning, and can realize large-scale real-time holographic imaging.

[0080] In order to implement the above-mentioned embodiment, the present invention also proposes an unsupervised deep learning holographic imaging device.

[0081] Figure 4 A schematic structural diagram of an unsupervised deep learning holographic imaging device provided in an embodiment of the present invention.

[0082] like Figure 4 As shown, the unsupervised deep learning holographic imaging device includes: an acquisition module 100, a construction module 200, and a training module 300, wherein:

[0083] An acquisition module, used for acquiring a hologram of a target to be trained;

[0084] A construction module, used for acquiring a corresponding holographic sub-image by propagating the hologram in free space, and constructing a data set including the hologram and the holographic sub-image;

[0085] The training module is used to construct a neural network whose input is a holographic sub-image and whose output is a hologram, and to train the neural network using a data set to obtain a trained neural network.

[0086] Furthermore, in one embodiment of the present invention, it also includes a reconstruction module, which is used to:

[0087] Collecting a hologram of the target to be restored;

[0088] The hologram is input into the trained neural network, and a reconstructed image of the target to be restored is output.

[0089] Furthermore, in one embodiment of the present invention, the building module is also used to:

[0090] Through computer simulation, the hologram is propagated a distance d in free space to obtain a holographic sub-image with a distance sample of 2d, and then a data set containing the hologram and the holographic sub-image is constructed.

[0091] Furthermore, in one embodiment of the present invention, the training module is also used to:

[0092] The hologram dataset is used as the reference data for the output of the neural network, and the hologram sub-image dataset is used as the input data for input into the network for training. The model parameters of the neural network are optimized through a predetermined number of forward and backward propagations.

[0093] Furthermore, in one embodiment of the present invention, the reconstruction module is further used to:

[0094] The image of the target to be restored is reconstructed through the nonlinear representation ability of the trained neural network and the redundancy of model information between holograms.

[0095] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0096] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0097] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. An unsupervised deep learning holographic imaging method, characterized in that: The following steps are involved: Collecting a hologram of the target to be trained; Acquire a corresponding hologram by propagating the hologram in free space, and construct a data set including the hologram and the hologram; Constructing a neural network whose input is the hologram and whose output is the hologram, training the neural network using the data set, and obtaining a trained neural network; in The method of obtaining a corresponding holographic sub-image by propagating the hologram in free space comprises: By means of computer simulation, the hologram is propagated in free space by a distance d to obtain a hologram sub-image with a distance sample of 2d, and then a data set including the hologram and the hologram sub-image is constructed; The using the data set to train the neural network includes: The hologram data set is used as the reference data of the output end of the neural network, and the hologram sub-image data set is input into the network as the input end data for training. The model parameters of the neural network are optimized through a predetermined number of forward and backward propagations.

2. The method according to claim 1, characterized in that Also includes: Collecting a hologram of the target to be restored; The hologram is input into the trained neural network, and a reconstructed image of the target to be restored is output.

3. The method according to claim 2, characterized in that The step of inputting the hologram into the trained neural network and outputting a reconstructed image of the target to be restored comprises: The image of the target to be restored is reconstructed through the nonlinear characterization capability of the trained neural network and the redundancy of model information between holograms.

4. An unsupervised deep learning holographic imaging device, characterized in that: Includes the following modules: An acquisition module, used for acquiring a hologram of a target to be trained; A construction module, used for acquiring a corresponding holographic sub-image by propagating the hologram in free space, and constructing a data set including the hologram and the holographic sub-image; A training module is used to construct a neural network whose input is the holographic sub-image and whose output is the hologram, and to train the neural network using the data set to obtain a trained neural network, wherein: The building blocks are also used to: By means of computer simulation, the hologram is propagated in free space by a distance d to obtain a hologram sub-image with a distance sample of 2d, and then a data set including the hologram and the hologram sub-image is constructed; The training module is also used for: The hologram data set is used as the reference data of the output end of the neural network, and the hologram sub-image data set is input into the network as the input end data for training. The model parameters of the neural network are optimized through a predetermined number of forward and backward propagations.

5. The device according to claim 4, characterized in that Also included is a reconstruction module for: Collecting a hologram of the target to be restored; The hologram is input into the trained neural network, and a reconstructed image of the target to be restored is output.

6. The device according to claim 5, characterized in that The reconstruction module is further used for: The image of the target to be restored is reconstructed through the nonlinear characterization capability of the trained neural network and the redundancy of model information between holograms.

Images