An Untrained Convolutional Neural Network Phase Retrieval Method for Sparse Fourier Stack Imaging

Abstract

The present invention belongs to the technical field of Fourier stack imaging, and specifically relates to a non-trained convolutional neural network phase recovery method applied to sparse Fourier stack imaging, comprising: acquiring a low-resolution image, performing interval sampling on the low-resolution image to obtain a low-resolution sub-image; performing interpolation amplification and adding random Gaussian noise to the low-resolution sub-image and then inputting the low-resolution sub-image into a convolutional neural network model to obtain the amplitude and phase information of the image, and reconstructing a low-resolution image according to the amplitude and phase information; comparing the difference between the reconstructed low-resolution image and the input low-resolution image, and training the neural network until the network converges to obtain the phase and amplitude information of the high-resolution image; the present invention designs a type of non-trained convolutional neural network, which can restore the image phase and amplitude information with high quality by using a small number of sampled image sequences without relying on a large amount of data.

Description

Technical Field

[0001] The present invention belongs to the technical field of Fourier stack imaging, and in particular relates to a non-trained convolutional neural network phase recovery method applied to sparse Fourier stack imaging. Background Art

[0002] The stack imaging technology proposed in recent years (which is divided into two categories: spatial stack imaging and Fourier stack imaging) uses synthetic aperture and phase retrieval technology to scan only a sub-area of ​​the sample or a low-resolution image of the sample each time, and solves the sample's high-resolution complex amplitude (including attenuation and phase information) optimization solution through computational imaging methods. Stack imaging technology can maintain the unity of large field of view and high resolution of image acquisition, while reconstructing the sample phase information and attenuation information (using specific algorithms to further reconstruct the sample depth information).

[0003] Fourier stack imaging technology uses synthetic aperture and phase retrieval technology to scan only a sub-area of ​​the sample or a low-resolution image of the sample each time, and solves the optimal solution of the high-resolution complex amplitude (including attenuation and phase information) of the sample through computational imaging methods. For example, the Fourier stack imaging reconstruction method based on deep learning is proposed by the series method proposed by Tian Lei's research group at Boston University. This method essentially uses data-driven training of a strongly supervised deep convolutional network to obtain the mapping relationship between the input low-resolution image sequence and the phase information of the object to be reconstructed, and finally obtains the reconstructed image. Although the Fourier imaging system maintains the unity of high resolution and large field of view, it takes a long time for the system to collect low-resolution images, which hinders the extension of the Fourier imaging system in terms of temporal resolution; and in the process of processing the image to be reconstructed, a large amount of data is required to train the model, which increases the difficulty and complexity of model training, and the accuracy of the amplitude and phase information of the high-resolution image reconstructed is poor. Summary of the invention

[0004] In order to solve the problems existing in the above prior art, the present invention proposes a non-trained convolutional neural network phase recovery method for sparse Fourier stack imaging, the method comprising: obtaining a sparse low-resolution image sequence to be reconstructed, performing an amplification and interpolation operation on the low-resolution image sequence, and adding random noise; inputting the low-resolution image sequence after adding random noise into a deep convolutional neural network to obtain the amplitude and phase information of the high-resolution image;

[0005] The process of training a deep convolutional neural network includes:

[0006] S1: Acquire a low-resolution image sequence, perform interval sampling on the low-resolution image sequence, and obtain a low-resolution sub-image sequence;

[0007] S2: enlarge the low-resolution sub-image sequence, and superimpose random Gaussian noise on the enlarged image;

[0008] S3: Inputting a low-resolution sub-image sequence with superimposed random noise into a deep convolutional neural network to obtain two feature maps, which correspond to the amplitude and phase information of the image to be reconstructed respectively; obtaining an object information map of the image to be reconstructed according to the amplitude and phase information of the image to be reconstructed;

[0009] S4: a low-resolution image reconstructed according to the object information map of the image to be reconstructed;

[0010] S5: constructing a system loss function according to the reconstructed low-resolution image and the input low-resolution image;

[0011] S6: When the loss function of the deep convolutional neural network converges, the low-resolution sequence is input to obtain the amplitude and phase information of the reconstructed high-resolution image, completing the training of the model.

[0012] Preferably, the enlarging operation of the low-resolution sub-image sequence includes: the size of the acquired low-resolution image sequence data is (Nx, Ny); the image size after enlarging the low-resolution sub-image sequence is (NX, NY), wherein NX=aNx, NY=aNY, and a is the scale coefficient.

[0013] Preferably, the deep convolutional neural network adopts a U-net structure.

[0014] Preferably, the formula for obtaining the object information graph of the image to be reconstructed according to the amplitude and phase information of the image to be reconstructed is:

[0015] V=Aexp(jθ)

[0016] Wherein, V represents the object information map of the image to be reconstructed, A represents the amplitude information of the image to be reconstructed, and θ represents the phase information of the image to be reconstructed.

[0017] Preferably, reconstructing a high-resolution image based on the object information map of the image to be reconstructed includes: obtaining the size of a low-resolution sub-image and the coordinates of the origin of the frequency domain center of the low-resolution sub-image; performing a two-dimensional Fourier forward transform on the object information map of the image to be reconstructed; performing a two-dimensional fftshift operation on the image that has undergone the two-dimensional Fourier forward transform; cropping the feature map that has undergone the two-dimensional fftshift operation based on the size of the low-resolution sub-image and the coordinates of the origin of the frequency domain center of the low-resolution sub-image to obtain a feature map of the same size as the input image; performing a two-dimensional fftshift operation on the cropped feature map again, and performing a two-dimensional Fourier inverse transform on the feature map that has undergone the two-dimensional fftshift operation again to obtain a reconstructed feature map; and processing the reconstructed feature map using a lens transfer function to obtain a reconstructed high-resolution image.

[0018] Furthermore, the formula for reconstructing a high-resolution image based on the object information map of the image to be reconstructed is:

[0019]

[0020] Among them, V represents the object information map of the image to be reconstructed, F and F -1 They represent the two-dimensional Fourier transform and inverse transform respectively, S represents the two-dimensional fftshift operation, and Crop represents the image cropping operation. y ) represents the coordinates of the frequency domain center origin of the low-resolution sub-image, (Nx, Ny) represents the size of the low-resolution sub-image, represents the convolution operation, and LF represents the lens transfer function.

[0021] Preferably, the system loss function is:

[0022]

[0023] Where V represents the object information map of the image to be reconstructed, N represents the number of low-resolution sub-images used to train the deep convolutional neural network, and I i (V) represents the reconstructed high-resolution image, J i Represents the input image.

[0024] Beneficial effects of the present invention:

[0025] In the process of Fourier stack imaging reconstruction, the present invention adopts a data-driven method to train the convolutional neural network, so that the phase information mapping relationship between the input image and the reconstructed image is obtained under a relatively small sampling image sequence, thereby obtaining high-quality restored image phase and amplitude information, thereby improving the training efficiency of the model. The present invention designs a type of untrained convolutional neural network, which can restore image phase and amplitude information with high quality without relying on a large amount of data and using a relatively small sampling image sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a structural diagram of the deep convolutional neural network of the present invention;

[0027] Figure 2 Amplitude and phase diagrams of experimental data reconstructed by the present invention and traditional methods. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] The present invention designs a type of untrained convolutional neural network, which can restore image phase and amplitude information with high quality by using a small number of sampled image sequences without the need for a large amount of collected data. The method includes: a Fourier stack imaging device performs sparse sampling to obtain a series of low-resolution collected images; a Fourier stack imaging forward model based on a convolutional neural network is established, and the collected low-resolution image sequence is enlarged and interpolated, and each image is added with random Gaussian noise, and then input into the convolutional neural network model, and the convolutional neural network outputs the predicted amplitude and phase information. After obtaining the predicted amplitude and phase values, a low-resolution image sequence is generated according to the information of the input low-resolution image sequence, such as the spectrum offset, etc., and the difference between the generated image sequence and the collected image sequence is compared, and the convolutional neural network is trained until the network converges.

[0030] A non-trained convolutional neural network phase recovery method for sparse Fourier stack imaging, such as the method comprising: obtaining a sparse low-resolution image sequence to be reconstructed, performing an amplification and interpolation operation on the low-resolution image sequence, and adding random noise; inputting the low-resolution image sequence after adding random noise into a deep convolutional neural network to obtain the amplitude and phase information of the high-resolution image.

[0031] In the existing Fourier stack imaging reconstruction process based on deep learning, the essence is to use data-driven training of deep convolutional networks to obtain the mapping relationship between the input low-resolution image sequence and the phase information of the object to be reconstructed, thereby obtaining the amplitude and phase information of the reconstructed high-resolution image. The present invention designs a type of untrained convolutional neural network, which can obtain high-quality restored image phase and amplitude information using a small number of sampled image sequences without relying on a large amount of data.

[0032] Figure 1 As shown in Figure 1, the process of training a deep convolutional neural network includes:

[0033] S1: Obtain low-resolution image sequence data, and the image size relative to the low-resolution images collected by all LEDs is recorded as (Nx, Ny); perform interval sampling on the low-resolution image sequence to obtain a number of low-resolution sub-image sequences;

[0034] The commonly used numbers of low-resolution image sequence data for training convolutional neural networks are 225 (15*15), 289 (17*17), and 441 (21*21). In the present invention, fewer low-resolution images are used for training, for example Figure 1 As shown, the present invention performs the above steps on the low-resolution image to obtain 25 low-resolution sub-images, and uses the 25 low-resolution sub-images to train the model, so that the phase and amplitude information of the high-quality restored image can be obtained by using a smaller sampling image sequence.

[0035] S2: Perform a zoom operation on a number of low-resolution sub-image sequences, and superimpose random Gaussian noise on the zoomed images.

[0036] S3: Input each low-resolution sub-image sequence superimposed with random noise into a deep convolutional neural network to obtain two feature maps, which correspond to the amplitude and phase information of the image to be reconstructed respectively; obtain the object information map of the image to be reconstructed according to the amplitude and phase information of the image to be reconstructed.

[0037] The constructed deep convolutional neural network uses a U-net similar structure, Leaky-ReLU is used as the activation unit, and the last layer uses a softmax layer output; the input size of the neural network is (NX, NY, N), and the output size is (NX, NY, 2).

[0038] S4: A low-resolution image reconstructed according to the object information map of the image to be reconstructed.

[0039] S5: constructing a system loss function according to the reconstructed low-resolution image and the input low-resolution image;

[0040] S6: When the loss function of the deep convolutional neural network converges, the low-resolution sequence is input to obtain the amplitude and phase information of the reconstructed high-resolution image, completing the training of the model.

[0041] The enlarging operation of the low-resolution sub-image sequence includes: the size of the acquired low-resolution image sequence data is (Nx, Ny); the image size after enlarging the low-resolution sub-image sequence is (NX, NY), wherein NX=aNx, NY=aNY, and a is the scale coefficient.

[0042] The formula for obtaining the object information graph of the image to be reconstructed based on the amplitude and phase information of the image to be reconstructed is:

[0043] V=Aexp(jθ)

[0044] Wherein, V represents the object information map of the image to be reconstructed, A represents the amplitude information of the image to be reconstructed, and θ represents the phase information of the image to be reconstructed.

[0045] Reconstructing a high-resolution image based on an object information map of an image to be reconstructed includes: obtaining the size of a low-resolution sub-image and the coordinates of the origin of the frequency domain center of the low-resolution sub-image; performing a two-dimensional Fourier forward transform on the object information map of the image to be reconstructed; performing a two-dimensional fftshift operation on the image that has undergone the two-dimensional Fourier forward transform; cropping a feature map that has undergone the two-dimensional fftshift operation based on the size of the low-resolution sub-image and the coordinates of the origin of the frequency domain center of the low-resolution sub-image to obtain a feature map of the same size as the input image; performing a two-dimensional fftshift operation on the cropped feature map again, and performing a two-dimensional Fourier inverse transform on the feature map that has undergone the two-dimensional fftshift operation again to obtain a reconstructed feature map; and processing the reconstructed feature map using a lens transfer function to obtain a reconstructed high-resolution image.

[0046] The formula for reconstructing a high-resolution image based on the object information map of the image to be reconstructed is:

[0047]

[0048] Among them, V represents the object information map of the image to be reconstructed, F and F -1 They represent the two-dimensional Fourier transform and inverse transform respectively, S represents the two-dimensional fftshift operation, and Crop represents the image cropping operation. y ) represents the coordinates of the frequency domain center origin of the low-resolution sub-image, (Nx, Ny) represents the size of the low-resolution sub-image, represents the convolution operation, and LF represents the lens transfer function.

[0049] The data constraint term of the system cost function proposed in the present invention is the minimum absolute error, and the stochastic gradient descent algorithm can be used for training, that is, the system loss function is:

[0050]

[0051] Where V represents the object information map of the image to be reconstructed, N represents the number of low-resolution sub-images used to train the deep convolutional neural network, and I i (V) represents the reconstructed high-resolution image, J i Represents the input image.

[0052] In order to measure the model performance, a sparse sampling Fourier stack imaging image reconstruction method based on a neural network of the present invention is compared with the classic GS reconstruction method. 25 128*128 low-resolution Fourier stack imaging experimental images are used, and the overlap of adjacent sub-apertures is only 18.1%. Figure 1 (a1) and (a2) are the reconstruction results of the full sampling of 441 images. Figure 2 (b1) and (b2) are traditional GS methods. Figure 1 (c1) and (c2) are reconstructed by the method of the present invention. From the results, it can be observed that the central part of the image (the area below the numbers 8 and 9) of the GS method is not correctly reconstructed, the numbers "5, 6" and the area next to them are not distinguished, the number "8" is blurred, and the phase cannot be clearly distinguished; the present invention correctly reconstructs both the phase and the amplitude, and the image is clear.

[0053] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A non-trained convolutional neural network phase recovery method for sparse Fourier stack imaging, characterized in that: include: Obtain a sparse low-resolution image sequence to be reconstructed, perform an enlargement and interpolation operation on the low-resolution image sequence, and add random noise; The low-resolution image sequence with random noise added is input into a deep convolutional neural network to obtain the amplitude and phase information of the high-resolution image; The process of training a deep convolutional neural network includes: S1: Acquire a low-resolution image sequence, perform interval sampling on the low-resolution image sequence, and obtain a low-resolution sub-image sequence; S2: enlarge the low-resolution sub-image sequence, and superimpose random Gaussian noise on the enlarged image; S3: Inputting a low-resolution sub-image sequence with superimposed random noise into a deep convolutional neural network to obtain two feature maps, which correspond to the amplitude and phase information of the image to be reconstructed respectively; obtaining an object information map of the image to be reconstructed according to the amplitude and phase information of the image to be reconstructed; S4: Reconstructing a low-resolution image according to the object information map of the image to be reconstructed; specifically comprising: obtaining the size of the low-resolution sub-image and the coordinates of the origin of the frequency domain center of the low-resolution sub-image; performing a two-dimensional Fourier forward transform on the object information map of the image to be reconstructed; performing a two-dimensional fftshift operation on the image that has undergone the two-dimensional Fourier forward transform; cropping the feature map that has undergone the two-dimensional fftshift operation according to the size of the low-resolution sub-image and the coordinates of the origin of the frequency domain center of the low-resolution sub-image to obtain a feature map of the same size as the input image; performing a two-dimensional fftshift operation on the cropped feature map again, and performing a two-dimensional Fourier inverse transform on the feature map that has undergone the two-dimensional fftshift operation again to obtain a reconstructed feature map; processing the reconstructed feature map using a lens transfer function to obtain a reconstructed high-resolution image; the formula for reconstructing a high-resolution image is: Among them, V represents the object information map of the image to be reconstructed, F and F -1 They represent the two-dimensional Fourier transform and inverse transform, S represents the two-dimensional fftshift operation, Crop represents the image cropping operation, (k x ,k y ) represents the coordinates of the frequency domain center origin of the low-resolution sub-image, (Nx, Ny) represents the size of the low-resolution sub-image, represents the convolution operation, LF represents the lens transfer function; S5: Construct a system loss function based on the reconstructed low-resolution image and the input low-resolution image; the expression of the loss function is: Where V represents the object information map of the image to be reconstructed, N represents the number of low-resolution sub-images used to train the deep convolutional neural network, and I i (V) represents the reconstructed high-resolution image, J i represents the input image; S6: When the loss function of the deep convolutional neural network converges, the low-resolution sequence is input to obtain the amplitude and phase information of the reconstructed high-resolution image, completing the training of the model.

2. The method for phase recovery using an untrained convolutional neural network for sparse Fourier stack imaging according to claim 1, characterized in that: The enlarging operation of the low-resolution sub-image sequence includes: the size of the acquired low-resolution image sequence data is (Nx, Ny); the image size after enlarging the low-resolution sub-image sequence is (NX, NY), wherein NX=aNx, NY=aNY, and a is the scale coefficient.

3. The method for phase recovery using an untrained convolutional neural network for sparse Fourier stack imaging according to claim 1, characterized in that: The deep convolutional neural network adopts the U-net structure.

4. The method for phase recovery using a non-trained convolutional neural network for sparse Fourier stack imaging according to claim 1, characterized in that: The formula for obtaining the object information graph of the image to be reconstructed based on the amplitude and phase information of the image to be reconstructed is: V=Aexp(jθ) Wherein, V represents the object information map of the image to be reconstructed, A represents the amplitude information of the image to be reconstructed, and θ represents the phase information of the image to be reconstructed.

Images