Holographic reconstruction method based on wavelet domain feature association complex valued neural network

Abstract

The invention belongs to the field of coaxial holographic reconstruction, and discloses a holographic reconstruction method based on a wavelet domain feature association complex-valued neural network, and the method specifically comprises the steps: adding a phase item to a coaxial hologram A to construct a complex amplitude B, carrying out the n-level discrete wavelet transformation of the complex amplitude B, taking the transformed result and the complex amplitude B as the input of the complex-valued neural network, and carrying out the n-level discrete wavelet transformation of the complex amplitude B; outputting object field complex amplitude C; performing noise suppression on the complex amplitude C of the object field, and simulating a coaxial holographic forward propagation process by using the suppressed complex amplitude D to generate a hologram E; and finally, calculating the similarity between the coaxial hologram A and the hologram E, carrying out smooth total variation constraint on the phase of the complex amplitude D, and realizing holographic reconstruction through a gradient descent optimization algorithm of a complex valued neural network. According to the method, the inherent coupling relation between the wavefront amplitude item and the phase item in the optical holography is utilized, the complex value feature correlation model is constructed to divide the target object and the background information, the target features are reserved, meanwhile, noise is restrained, and more excellent holographic reconstruction is achieved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of on-axis holographic reconstruction, and more particularly relates to a holographic reconstruction method based on a wavelet domain feature correlation complex-valued neural network. BACKGROUND

[0002] On-axis holography is a holographic imaging technology based on the premise of coherent light interference theory, which uses the coherent superposition of reference light and object light to produce an interference pattern, and selects a camera to record it. The recorded interference pattern contains the amplitude and phase information of the target, and can realize the three-dimensional structure reconstruction of the target object. Due to the advantages of compact structure, economy and efficiency, non-contact, etc., it has been widely used in optical imaging, material microstructure analysis, biomedical detection and data storage, etc. With the development of modern science and technology, integrating image processing algorithms and deep learning into on-axis holographic reconstruction can realize the expression of more complex physical information and more excellent reconstruction effect, and has greater potential in many fields such as three-dimensional dynamic imaging of cells in biomedicine and high-precision detection of industrial instruments, and has become an important tool for many scientists to study and apply.

[0003] However, due to the inevitable factors such as noise influence in the process of on-axis holographic interference pattern shooting and turbulent disturbance, the target information is mixed with the background information in the hologram, which makes the ill-posed inverse process of holographic reconstruction more difficult. The existing method directly forces the denoising of the reconstructed phase, which may cause the loss of target features.

[0004] The complex-valued neural network can better express the coupling relationship between the amplitude term and the phase term of the wavefront in the optical diffraction propagation by directly calculating the complex number, and can provide more rich feature expression ability than the real-valued neural network in the process of on-axis holographic recovery. Since the noise or disturbance term exists in the amplitude term and the phase term in the process of holographic imaging, constructing a denoising model or edge detection model based on complex values can bring more stable noise suppression ability and more robust inter-layer feature information transmission effect, thereby helping to improve the information representation ability of holographic imaging. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a holographic reconstruction method based on a wavelet domain feature correlation complex-valued neural network, which aims to utilize the inherent coupling relationship between the amplitude term and the phase term of the wavefront in optical holography, construct a complex feature correlation model to divide the target object and the background information, retain the target features while suppressing the noise, and realize more excellent holographic reconstruction.

[0006] The technical solution of the present application to solve the above technical problems is:

[0007] A holographic reconstruction method based on a wavelet domain feature correlation complex-valued neural network, comprising the following steps:

[0008] S1, build an optical path based on the principle of coaxial holography and use a camera to shoot a phase type coaxial hologram A;

[0009] S2, square the pixel value of the collected phase type coaxial hologram A as the amplitude term, and randomly generate an initial phase term to construct a complex amplitude B. After n-level discrete wavelet transform of the complex amplitude B, the low-frequency complex amplitude components and high-frequency complex amplitude components at each level are generated, which are used as the input of the complex-valued network model together with the original complex amplitude B. The phase term of the suppressed complex amplitude D output by each iteration will replace the phase term of the input complex amplitude B to generate a new complex amplitude B;

[0010] S3, the input of the complex-valued network model is first subjected to preliminary feature extraction by a shallow feature extraction layer composed of multiple complex convolution blocks with a kernel size of 3x3. Each complex convolution block contains a complex convolution operation, a complex batch normalization layer, and a complex ReLU activation function. Then, a complex Prewitt operator module is used to construct edge masks for the low-frequency components at each level output by the shallow feature extraction layer, enhancing the weight of the edge features of the low-frequency components. Subsequently, the enhanced low-frequency components and the corresponding high-frequency components are used as the input of the cross-wavelet domain feature association module.

[0011] S4, in the cross-wavelet domain feature association module, four complex deep separable convolutions are used to calculate Q, V L and K, V H of the enhanced low-frequency components and the corresponding high-frequency components, respectively. The weights W LH of the low-frequency text information focused in the high-frequency components and the weights W HL of the low-frequency components on the semantic expression of the high-frequency components are calculated using multi-head attention and a softmax function. Finally, the weights W LH and V L , and the weights W HL and V H are combined and applied to the input enhanced low-frequency components and the corresponding high-frequency components, respectively, to obtain enhanced low-frequency features ENL and high-frequency features ENH.

[0012] S5, the n-level low-frequency features ENL n and high-frequency features ENH n are subjected to inverse discrete wavelet transform and a complex convolution block. Then, the output of the complex convolution block is combined with the n-1-level low-frequency features ENL n-1 to generate optimized low-frequency features ENL ' n-1 ; the optimized low-frequency features ENL ' n-1 are combined with the corresponding high-frequency features ENH n-1The inverse discrete wavelet transform is performed and a complex value convolution block is passed, and n same operations are performed to generate a complex amplitude C of the object field with the same size as the complex amplitude B;

[0013] S6, noise suppression is performed on the complex amplitude C of the object field using a complex value based smoothing total variation denoising model, and the suppressed complex amplitude D is simulated coaxial holographic forward propagation to generate a hologram E; finally, the similarity between the hologram phase type coaxial hologram A and the hologram E is calculated, and the phase of the complex amplitude D is smoothed total variation constrained, and the gradient descent optimization algorithm of the complex value neural network is used for single image self-supervised iterative optimization process, and finally the holographic reconstruction is realized.

[0014] Preferably, in step S1, the phase type coaxial hologram A is a pure phase hologram, and the amplitude of the original object wave is set to 1.

[0015] Preferably, in step S2, the size of the randomly generated initial phase term is the same as that of the phase type coaxial hologram A, and the range is 0 to 1.

[0016] Preferably, in step S2, the n-level discrete wavelet transform uses two-dimensional Haar discrete wavelet transform, and outputs low frequency component LL, high frequency horizontal component HL, high frequency vertical component LH and high frequency diagonal component HH; n corresponds to the number of times of two-dimensional Haar discrete wavelet transform; the complex amplitude B is used as the first level low frequency component for two-dimensional Haar discrete wavelet transform, and the output size of the low frequency component LL1, the high frequency horizontal component HL1, the high frequency vertical component LH1 and the high frequency diagonal component HH1 is 1 / 4 of the original input size; each level of two-dimensional Haar discrete wavelet transform only operates on the low frequency component, and the size of each component output is 1 / 4 of the input size of this level; the input of the complex value network model includes the input complex amplitude B and all high frequency components and low frequency components generated at each level.

[0017] Preferably, in step S3, the complex Prewitt operator module uses the horizontal and vertical 3x3 convolution templates of the Prewitt operator as the convolution kernel to perform neighborhood convolution operation on the multi-channel features composed of the real part and the imaginary part of the complex amplitude, and calculate the spatial gradient changes of the real part and the imaginary part respectively; create an edge mask (mask value between 0 and 1, edge region close to 1, non-edge region close to 0) after normalization of the gradient changes; modulate (mask and real part, imaginary part product) the real part and the imaginary part through the edge mask, and then superimpose the modulation results with the original real part and imaginary part respectively to realize the optimization of the edge enhanced complex value signal.

[0018] Preferably, in step S5, the complex amplitude C of the object field needs to be additionally passed through a complex value convolution layer with one channel contraction. The complex value convolution layer only includes one complex value convolution operation.

[0019] Preferably, in step S6, the complex-valued smoothing total variational denoising model incorporates the complex-valued total variation into a Huber function, and constructs a Huber-TV regularization layer based on the iterative steps of the proximal gradient method using this Huber function; the proximal operator is approximated using gradient descent, wherein the gradient update step size s is selected as 0.1; the complex amplitude generated iteratively is compared with the input complex amplitude to ensure data fidelity, so that the gradient direction is pulled back towards the direction of data consistency, avoiding over-smoothing; the number of iterations a of the Huber-TV regularization layer is set in the range of 5-10 times.

[0020] Preferably, in step S6, the smoothing total variation constraint is a real-valued constraint on the phase term, which is composed of the real-valued total variation incorporated into a differentiable Huber function, and a hyperparameter α is set to prevent the constraint from becoming too smooth. The range of α can be from 0 to 1.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] 1. The holographic reconstruction method based on wavelet domain feature correlation complex value neural network of the present invention constructs a Prewitt operator module based on complex values, convolves the Prewitt operator on multi-channel complex value features, calculates the gradient magnitudes of the real and imaginary parts of the complex number in the horizontal and vertical directions respectively, enhances the edge feature expression by constructing edge weight masks for the real and imaginary parts, and fuses the Prewitt operator processing results with the original feature map to realize edge determination in the complex domain. The sensitivity to low-frequency features is improved by low-frequency enhancement and complex processing.

[0023] 2. The holographic reconstruction method based on wavelet domain feature association complex-valued neural networks of the present invention achieves the extraction of low-frequency text information focused by high-frequency components and the enhancement of the weights expressing high-frequency detail information in low-frequency components by constructing a cross-wavelet domain feature association module that explores the correlation between low-frequency and high-frequency components. This cross-wavelet domain feature association module uses the low-frequency components and the sum of each high-frequency component as dual inputs, and uses four complex-valued depthwise separable convolutions to calculate the feature representations Q and V of the low-frequency components respectively. L and the characteristic representations of high-frequency components K and V H By combining multi-head attention and the softmax function, the gain of high-frequency components on low-frequency components and low-frequency components on high-frequency components is calculated, and this gain is superimposed on the feature components of the original input to achieve the correlation between cross-component features.

[0024] 3. The holographic reconstruction method based on the wavelet domain feature correlation complex-valued neural network of the application realizes the derivability of the model in the regularization constraint of the phase item in the back propagation process by integrating the total variation into a differentiable Huber function; further, the Huber-TV regularization layer is constructed by using the iteration steps of the proximal gradient method for the fused function, the proximal operator is approximated by using the gradient descent, the complex amplitude suppressing noise is iteratively generated, and the generated complex amplitude and the input complex amplitude are calculated for data fidelity, so that the gradient direction is pulled back to the data consistency direction, and the excessive smoothing of the generated complex amplitude caused by the noise suppression is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The flow block diagram of the holographic reconstruction method based on the wavelet domain feature correlation complex-valued neural network of the application.

[0026] Figure 2 The algorithm framework diagram of the holographic reconstruction method based on the wavelet domain feature correlation complex-valued neural network of the application. DETAILED DESCRIPTION

[0027] The application will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the application are not limited thereto.

[0028] Reference Figures 1-2 The holographic reconstruction method based on the wavelet domain feature correlation complex-valued neural network of the application includes the following steps:

[0029] S1. An optical path is built on the basis of the principle of on-axis holography, and a phase-type on-axis hologram A is photographed by using a camera;

[0030] S2. The pixel value of the collected phase-type on-axis hologram A is squared as an amplitude item, and an initial phase item is randomly generated to construct a complex amplitude B. After the complex amplitude B is subjected to n-level discrete wavelet transform, low-frequency complex amplitude components and high-frequency complex amplitude components are generated, which are used as inputs of the complex-valued network model together with the original complex amplitude B. The phase item of the suppressed complex amplitude D output in each iteration replaces the phase item of the input complex amplitude B to generate a new complex amplitude B;

[0031] S3. The inputs of the complex-valued network model are all subjected to preliminary feature extraction by a shallow feature extraction layer, which is composed of multiple complex-valued convolution blocks with a convolution kernel size of 3x3. Each complex-valued convolution block includes a complex-valued convolution operation, a complex-valued batch normalization layer, and a complex-valued ReLU activation function. Then, a complex-valued Prewitt operator module is used to construct an edge mask of each low-frequency component output by the shallow feature extraction layer to enhance the weight of the edge feature of the low-frequency component. Subsequently, the enhanced low-frequency component and the corresponding high-frequency component are used as inputs of a cross-wavelet domain feature correlation module.

[0032] S4. In the cross-wavelet domain feature association module, four complex-valued depthwise separable convolutions are used to calculate the Q and V values ​​for enhancing the low-frequency components. L And the corresponding high-frequency components K and V H The weight W of the low-frequency text information focused in the high-frequency components is calculated by combining multi-head attention and the softmax function. LH And the weight W of the low-frequency component on the semantic representation of the high-frequency component. HL Finally, the weight W LH With V L Weight W HL With V H By combining and applying the enhanced low-frequency component and the corresponding high-frequency component to the input respectively, the enhanced low-frequency feature ENL and the high-frequency feature ENH are obtained.

[0033] S5, ENL of the nth level low-frequency feature n and high-frequency characteristics ENH n Perform a discrete wavelet inverse transform and then pass it through a complex-valued convolutional block. Finally, combine the output of the complex-valued convolutional block with the (n-1)th level low-frequency feature ENL. n-1 The combination generates an optimized low-frequency feature ENL. ' n-1 ;Optimized low-frequency features ENL ' n-1 Then, with the corresponding high-frequency characteristic ENH n-1 After performing discrete wavelet inverse transform and a complex convolution block, and executing the same operation n times, a complex amplitude C of the object field with the same size as the complex amplitude B is generated.

[0034] S6. Noise suppression is achieved by using a complex-valued smoothing total variational denoising model for the complex amplitude C of the object field. The suppressed complex amplitude D is then used to simulate the forward propagation process of a coaxial hologram to generate a hologram E. Finally, the similarity between the phase-type coaxial hologram A and the hologram E is calculated, and the phase of the complex amplitude D is subject to smoothing total variational constraints. A self-supervised iterative optimization process for a single image is then performed using the gradient descent optimization algorithm of a complex-valued neural network, ultimately achieving holographic reconstruction.

[0035] See Figure 1 In step S1, the phase-type coaxial hologram A is a pure phase hologram, and the amplitude of the original object wave is set to 1.

[0036] See Figures 1-2 In step S2, the size of the randomly generated initial phase term is the same as that of the hologram A, ranging from 0 to 1.

[0037] See Figures 1-2In step S2, the n-level discrete wavelet transform uses a two-dimensional Haar discrete wavelet transform, and outputs a low-frequency component LL, a high-frequency horizontal component HL, a high-frequency vertical component LH, and a high-frequency diagonal component HH; n of the n-level discrete wavelet transform corresponds to the number of times of using the two-dimensional Haar discrete wavelet transform; the complex amplitude B is subjected to the two-dimensional Haar discrete wavelet transform as a first-level low-frequency component, and a low-frequency component LL1, a high-frequency horizontal component HL1, a high-frequency vertical component LH1, and a high-frequency diagonal component HH1 with a size of 1 / 4 of the original input size are output; each subsequent two-dimensional Haar discrete wavelet transform is only performed on a low-frequency component, and each component output has a size of 1 / 4 of the input size of this level; the input of the complex-valued network model includes the input complex amplitude B and all high-frequency components and low-frequency components generated at each level.

[0038] Referring to Figures 1-2 In step S2, n of the n-level discrete wavelet transform corresponds to the number of times of using the two-dimensional Haar discrete wavelet transform, and in this embodiment, the input complex amplitude has a size of 256x256, and n is set to 3, so that the complex amplitude has sizes of 128x128, 64x64, and 32x32 after three discrete wavelet samplings.

[0039] Referring to Figure 2 The input channel number of all complex convolution blocks for convolving the initial input multi-level complex amplitude is 1, and the output channel number is 64; the input channel number of the complex convolution block after channel splicing is 128, and the output channel number is 64, and the input and output channel numbers of the complex convolution blocks in the remaining intermediate processes are all 64; a larger channel number can enhance the learning ability of the model for complex amplitude features, but requires larger memory operation and increases the time cost of model learning; the generated object field complex amplitude C needs to pass through a complex convolution layer with a channel number of 1, and the complex convolution layer only includes one complex convolution operation.

[0040] Referring to Figure 1 In step S6, the Huber-TV regularization layer uses gradient descent approximation to approximate a proximal operator, and uses an iterative optimization denoising method to suppress noise, and in this embodiment, the iteration number a is set to 7, and the gradient update step s is set to 0.1.

[0041] Referring to Figure 2The loss constructed in the embodiment is composed of two parts, the first part is to calculate the mean square error between the intensity of the input hologram A and the intensity of the generated hologram E, and p and q represent the length and width of the hologram respectively; the second part is to perform Huber-TV smoothing total variation constraint on the phase term in the complex amplitude D, wherein the smoothing total variation constraint is composed of a differentiable Huber function by fusing a real total variation; wherein the hyperparameter alpha is used to prevent excessive smoothing of the phase term, and a larger alpha weight value can be assigned to an image with larger noise, and the value of the hyperparameter alpha is set to 0.1 in the embodiment.

[0042] Referring to Figures 1-2 In step S6, the propagation method of generating hologram E by simulating the coaxial holographic forward propagation process of the inhibited complex amplitude D is an angular spectrum propagation method in the embodiment.

[0043] The above is the preferred embodiment of the present application, but the embodiments of the present application are not limited by the above, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, which are all included in the protection scope of the present application.

Claims

1. A holographic reconstruction method based on a complex-valued neural network associated with wavelet domain features, characterized in that, The method comprises the following steps: S1, on the basis of the coaxial holographic basic principle, an optical path is built and a phase type coaxial hologram A is photographed by a camera; S2, the pixel value of the collected phase type coaxial hologram A is squared as an amplitude item, and an initial phase item is randomly generated to construct a complex amplitude B, the complex amplitude B is subjected to n-level discrete wavelet transform to generate low-frequency complex amplitude components and high-frequency complex amplitude components at each level, which are used as inputs of a complex value network model together with the original complex amplitude B; the phase item of the suppressed complex amplitude D output by each iteration is used to replace the phase item of the input complex amplitude B to generate a new complex amplitude B; S3, the inputs of the complex value network model are all subjected to preliminary feature extraction through a shallow feature extraction layer, the shallow feature extraction layer is composed of a plurality of complex value convolution blocks with a convolution kernel size of 3*3, each complex value convolution block comprises a complex value convolution operation, a complex value batch normalization layer and a complex value ReLU activation function; then, an edge mask of each low-frequency component output by the shallow feature extraction layer is constructed by using a complex value Prewitt operator module to enhance the weight of the edge feature of the low-frequency component; subsequently, the enhanced low-frequency component and the corresponding high-frequency component are used as inputs of a cross wavelet domain feature correlation module; S4, in the cross wavelet domain feature correlation module, using four complex depth separable convolution respectively to calculate the enhanced low frequency component Q, V L And the corresponding high frequency component K, V H , combined with multi-head attention and softmax function to calculate the weight W of the focused low frequency text information in the high frequency component LH And the weight W of the low frequency component to the semantic expression of the high frequency component HL ; finally, the weight W LH And V L , the weight W HL And V H Combined and respectively acted on the input enhanced low frequency component and the corresponding high frequency component, to obtain the enhanced low frequency feature ENL and high frequency feature ENH; S5, the n-level low-frequency feature ENL n and high-frequency feature ENH n discrete wavelet inverse transform and a complex convolution block, and then combine the output of the complex convolution block with the n-1-level low-frequency feature ENL n-1 to generate an optimized low-frequency feature ENL ' n-1 ; the optimized low-frequency feature ENL ' n-1 and the corresponding high-frequency feature ENH n-1 discrete wavelet inverse transform and a complex convolution block, and after performing the same operation n times, a complex amplitude C of the object field with the same size as the complex amplitude B is generated. S6, a complex value based smoothing total variation denoising model is used for noise suppression on the object field complex amplitude C, the suppressed complex amplitude D is used to simulate a coaxial holographic forward propagation process to generate a hologram E; finally, the similarity between the hologram phase type coaxial hologram A and the hologram E is calculated, and the phase of the complex amplitude D is subjected to smoothing total variation constraint, and a single image self-supervised iterative optimization process is performed through a gradient descent optimization algorithm of the complex value neural network, so that holographic reconstruction is finally realized.

2. The holographic reconstruction method based on wavelet domain feature correlation complex-valued neural network according to claim 1, characterized in that, In step S1, the phase type coaxial hologram A is a pure phase hologram, and the amplitude of the original object wave is set to 1.

3. The holographic reconstruction method based on wavelet domain feature correlation complex-valued neural network according to claim 1, characterized in that, In step S2, the size of the randomly generated initial phase item is the same as that of the phase type coaxial hologram A, and the range is 0 to 1.

4. The holographic reconstruction method based on wavelet domain feature correlation complex-valued neural network according to claim 1, characterized in that, In step S2, the n-level discrete wavelet transform adopts two-dimensional Haar discrete wavelet transform, and outputs low-frequency components LL, high-frequency horizontal components HL, high-frequency vertical components LH and high-frequency diagonal components HH; n corresponds to the number of times of two-dimensional Haar discrete wavelet transform; The complex amplitude B is subjected to two-dimensional Haar discrete wavelet transform as the first low-frequency component, and outputs a low-frequency component LL1, a high-frequency horizontal component HL1, a high-frequency vertical component LH1 and a high-frequency diagonal component HH1 with a size of 1 / 4 of the original input size; each two-dimensional Haar discrete wavelet transform only operates on the low-frequency component, and the size of each component output is 1 / 4 of the input size at this level; the inputs of the complex value network model include the input complex amplitude B and all high-frequency components and low-frequency components generated at each level.

5. The holographic reconstruction method based on wavelet domain feature correlation complex-valued neural network according to claim 1, characterized in that, In step S3, the complex Prewitt operator module uses the horizontal and vertical 3*3 convolution templates of the Prewitt operator as a convolution kernel to perform neighborhood convolution operation on the multi-channel features composed of the real part and the imaginary part of the complex amplitude, and calculate the spatial gradient changes of the real part and the imaginary part, respectively; create an edge mask after normalizing the gradient changes; modulate the real part and the imaginary part through the edge mask, and then superimpose the modulation results with the original real part and the original imaginary part, respectively, to realize the optimization of the edge-enhanced complex value signal.

6. The holographic reconstruction method based on wavelet domain feature correlation complex-valued neural network according to claim 1, characterized in that, In step S5, the complex amplitude C of the object field needs to pass through an additional complex value convolution layer with one channel contraction to 1, and the complex value convolution layer only contains one complex value convolution operation.

7. The holographic reconstruction method based on wavelet domain feature correlation complex-valued neural network according to claim 1, characterized in that, In step S6, the complex-based smooth total variation denoising model integrates the complex total variation into a Huber function, and constructs a Huber-TV regularization layer by using the iterative steps of the proximal gradient method on the basis of the Huber function; the gradient descent is used to approximate the proximal operator, wherein the gradient update step s is selected as 0.1; the generated complex amplitude in iteration is data-faithful to the input complex amplitude, so that the gradient direction is pulled back to the data consistency direction, and over-smoothing is avoided.

8. The holographic reconstruction method based on wavelet domain feature correlation complex-valued neural network according to claim 7, characterized in that, The setting range of the iteration number a of the Huber-TV regularization layer is 5-10 times.

9. The holographic reconstruction method based on wavelet domain feature correlation complex-valued neural network according to claim 1, characterized in that, In step S6, the smooth total variation constraint is a real value constraint for the phase term, which is composed of a Huber function by integrating the real value total variation.

Images