Method and device for realizing multiplexing Fourier laminated imaging based on prior neural network

Abstract

The invention relates to a method for realizing multiplexing Fourier laminated imaging based on a prior neural network. The method comprises the following steps: acquiring an initial image from a data set; performing restoration processing on the phase of the initial image to obtain a phase restoration image; restoring the amplitude of the initial image to obtain an amplitude restored image; synthesizing the phase restoration image and the amplitude restoration image to obtain a complex amplitude image; performing forward Fourier transform processing on the complex amplitude diagram to obtain a low-definition diagram; performing frequency shift processing on the low-definition image to obtain a light display image; and performing inverse Fourier transform processing on the light display image to obtain a high-definition image. By adopting the method for realizing multiplexing Fourier laminated imaging based on the prior neural network, the acquisition efficiency can be improved while the quality of a reconstructed image is ensured.

Description

Technical Field

[0001] The present invention relates to the field of neural networks, and particularly to a method for implementing multiplex Fourier ptychography based on a neural network. Background Art

[0002] Fourier ptychography (FP) replaces the light source of an optical microscope with an LED array and adds the NA of the light source expansion system to obtain an improvement in the resolution of the optical microscope. At the same time, phase information is also obtained, and the resolution can be increased by several times. Fourier ptychography has now been widely applied in biomedical fields such as digital pathology analysis, cell and tissue imaging, and polarization-sensitive imaging.

[0003] In the forward process of Fourier ptychography, multiple low-resolution (LR) images are obtained by tilting illumination with multiple lights. These LR images contain the consistency information contained in the high-resolution (HR) image, but these LR images need to be processed by computational imaging to obtain the HR image, such as iterative convergence and deep learning. Traditional iterative algorithms rely heavily on the initial amplitude and phase, and the reconstructed amplitude and phase are poor. Supervised neural networks can reconstruct the amplitude and phase well, but generally have poor generality and require a large amount of data sets.

[0004] Please refer to the patent with the publication number "CN111650738A", which shows a method of using an unsupervised network for model calculation. After collecting a single sample, initialization and iterative optimization are performed to reconstruct the image to obtain a reconstructed image, thereby reducing the demand for the sample data set. However, this method needs to calculate the frequency domain difference, and the frequency domain has sparsity, that is, only a small part of the spectrogram has large value points, and the values of the rest are close to 0. This sparsity will not only increase the computational complexity, but may also lead to a decline in the model performance, and the acquisition efficiency is low. Usually, more than 200 images need to be collected for a single sample and the corresponding LED incident wave vector sequence needs to be recorded, which requires a lot of time, manpower and material resources and cannot meet the actual needs. Summary of the Invention

[0005] Based on this, the object of the present invention is to provide a method for implementing multiplex Fourier ptychography based on a prior neural network, which can directly calculate the difference between low-resolution images, can achieve higher resolution and higher phase contrast only using a small amount of data sets, and greatly improves the acquisition efficiency by using a multiplexing scheme.

[0006] A method for implementing multiplex Fourier ptychography based on a prior neural network includes:

[0007] Collecting an initial image from a data set;

[0008] Performing phase restoration processing on the phase of the initial image to obtain a phase-restored image;

[0009] The amplitude of the initial image is restored to obtain an amplitude-restored image;

[0010] The phase-restored image and the amplitude-restored image are combined to obtain a complex amplitude image;

[0011] The complex amplitude image is subjected to a forward Fourier transform to obtain a low-resolution image;

[0012] The low-resolution image is frequency-shifted to obtain a light display image;

[0013] The light display image is subjected to an inverse Fourier transform to obtain a high-resolution image.

[0014] Further, in the frequency shift processing, the pupil function is:

[0015]

[0016] where NA is the numerical aperture, λ is the light wavelength, and (k x , k y ) is a two-dimensional frequency domain coordinate system.

[0017] Further, it also includes: obtaining a dataset for the sample by using a multi-lamp multiplexing method; taking multiple pictures by changing different illumination directions and quantities to obtain a dataset; after the Fourier forward process of the sample collected by multi-lamp multiplexing satisfies:

[0018]

[0019] where I p represents the light intensity captured under the simultaneous illumination of multiple lights, r represents the coordinates in the sample space, and I m represents the light intensity captured under the illumination of a single light, represents the inverse Fourier transform, and O(k - k m ) represents the shift of the spectrum after the sample is illuminated by the tilted light.

[0020] Further, it also includes: calculating a loss function based on the initial image and the high-resolution image to update the restoration processing parameters;

[0021] The loss function (loss) is:

[0022]

[0023] where I j is the amplitude of the initially collected image, I p is the amplitude of the high-resolution image predicted by the network, and combined with the formula satisfied by the sample in the forward process of Fourier ptychography, we get

[0024]

[0025] Further, it also includes: continuously checking the high-definition image and the number of updates, and outputting a reconstructed image when the set standard is met.

[0026] The present invention also provides a multiplexed Fourier ptychography imaging device based on a prior neural network, including:

[0027] An acquisition module for acquiring an initial image from a data set;

[0028] A phase restoration module for performing phase restoration processing on the phase of the initial image to obtain a phase-restored image;

[0029] An amplitude restoration module for performing amplitude restoration processing on the amplitude of the initial image to obtain an amplitude-restored image;

[0030] A synthesis module for synthesizing the phase-restored image and the amplitude-restored image to obtain a complex amplitude image;

[0031] A Fourier transform module for performing forward Fourier transform processing on the complex amplitude image to obtain a low-definition image;

[0032] A frequency shift module for performing frequency shift processing on the low-definition image to obtain an optically displayed image;

[0033] An inverse Fourier transform module for performing inverse Fourier transform processing on the optically displayed image to obtain a high-definition image.

[0034] Further, in the frequency shift module, the pupil function adopted is:

[0035]

[0036] where NA is the numerical aperture, λ is the optical wavelength, and (k x ,k y ) is a two-dimensional frequency domain coordinate system.

[0037] Further, it also includes a light source and a sensor. The data set of the sample is obtained by using a multi-lamp multiplexing method; the different illumination directions and numbers of the light source are changed, and the sensor takes multiple pictures to obtain the data set; after multi-lamp multiplexing acquisition, the forward Fourier process of the sample satisfies:

[0038]

[0039] where I p represents the light intensity captured by the sensor under the simultaneous illumination of multiple light sources, r represents the coordinate in the sample space, and I m represents the light intensity captured by the sensor under the illumination of a single light source, represents the inverse Fourier transform, and O(k - k m ) represents the shift of the spectrum of the sample after being illuminated by the tilted light.

[0040] Further, it further includes an updating module, configured to calculate a loss function based on the initial image and the high-definition image to update the restoration processing parameters;

[0041] The loss function (loss) is:

[0042]

[0043] where I j is the amplitude of the initially acquired image, and I p is the amplitude of the high-definition image predicted by the network. It is obtained by combining the formula satisfied by the samples in the forward process of Fourier ptychography

[0044]

[0045] Further, it further includes an inspection module, configured to continuously inspect the high-definition image and the number of update times, and output a reconstructed image when the set standard is met.

[0046] For better understanding and implementation, the present invention will be described in detail below with reference to the accompanying drawings. Description of the Drawings

[0047] Figure 1 It is a schematic structural diagram of the multiplexed Fourier ptychography imaging device based on the prior neural network of the present invention.

[0048] Figure 2 It is a flowchart of the multiplexed Fourier ptychography imaging method based on the prior neural network of the present invention.

[0049] Figure 3 It is a phase contrast diagram of the reconstructed images of the multiplexed Fourier ptychography imaging method based on the prior neural network of the present invention and the existing Fourier ptychography imaging reconstruction method.

[0050] Figure 4 It is a phase contrast diagram of the reconstructed images of the multiplexed Fourier ptychography imaging method based on the prior neural network of the present invention and the existing Fourier ptychography imaging reconstruction method using the single-lamp algorithm and multi-lamp multiplexing respectively. Detailed Embodiment

[0051] The applicant carefully analyzed the existing Fourier ptychography imaging reconstruction method and found that the reason for its low acquisition efficiency is that it is applied to the single-lamp algorithm, which requires continuous replacement of the light source and position for shooting, resulting in low acquisition efficiency of the data set. Therefore, the present invention attempts to use a multi-lamp multiplexing scheme for acquisition. In order to adapt to the large amount of data obtained by multi-lamp multiplexing, the operation logic of the neural network is changed, the difference of the low-resolution image is directly calculated, and a constant pupil function is added to stabilize the image frequency shift calculation to ensure the subsequent image reconstruction quality.

[0052] Based on this, please refer to Figure 1, the present invention designs a multiplexed Fourier ptychography imaging device based on a prior neural network. Please refer to Figure 2 , Figure 2 Figure shows the operation flow of the multiplexed Fourier ptychography imaging device based on a prior neural network of the present invention.

[0053] The multiplexed Fourier ptychography imaging device based on a prior neural network of the present invention includes a light source 10 (not shown in the figure), a sensor 20 (not shown in the figure), an acquisition module 30, a phase restoration module 51, an amplitude restoration module 52, a synthesis module 60, a Fourier transform module 70, a frequency shift module 80, an inverse Fourier transform module 90, a verification module 100, and an update module 40.

[0054] The light source 10 includes an LED board, on which a plurality of identical LED lights are provided, and the on / off of any one LED light can be freely controlled. The light source 10 is arranged on one side of the sample, and the light beam emitted by the LED lights can transmit through the sample to be collected.

[0055] The sensor 20 is arranged on the other side of the sample to receive the light beam transmitted through the sample by the light source 10, and a plurality of sample images are obtained to form a data set.

[0056] It can be understood that when using the existing multi-light multiplexing method to obtain a data set for the sample, first turn on a plurality of LEDs on the LED board of the light source 10, take a picture through the sensor 20, and then turn on a different several LEDs to take another picture after shooting until all the pictures in the required data set are taken to obtain the data set. Specifically, when the total number of lights is N*M, first randomly turn on M lights to collect a photo, then remove these M lights, and randomly turn on M lights from the remaining N*M - M lights to collect a photo until each LED light has been turned on at least once, and finally a data set with N low-resolution images is collected.

[0057] The acquisition module 30 acquires an initial image from the data set. Specifically, the acquisition module 30 randomly acquires an initial image from multiple images in the data set. In particular, random noise can also be used instead of the initial image as the input to obtain the corresponding noise reconstruction image.

[0058] The phase restoration module 51 performs phase restoration processing on the phase of the initial image to obtain a phase restoration image. The phase restoration module 51 includes a plurality of upsampling layers and a plurality of downsampling layers, which perform feature selection on the phase of the initial image to reduce the actual computational amount, and the number of upsampling layers and downsampling layers corresponds. Specifically, 4 upsampling layers and 4 downsampling layers are set for feature selection.

[0059] The amplitude restoration module 52 performs amplitude restoration processing on the amplitude of the initial image to obtain an amplitude-restored image. Similarly, the amplitude restoration includes multiple upsampling layers and multiple downsampling layers, which perform feature selection on the amplitude of the initial image to reduce the actual computational amount, where the number of upsampling layers and downsampling layers corresponds. Specifically, 4 upsampling layers and 4 downsampling layers are set for the feature selection module.

[0060] The synthesis module 60 synthesizes the phase-restored image and the amplitude-restored image to obtain a complex amplitude image. The synthesis module 60 synthesizes the complex amplitude distribution of the sample according to the phase distribution in the phase-restored image and the amplitude distribution in the amplitude-restored image, and constructs a complex amplitude image according to the complex amplitude distribution.

[0061] The Fourier transform module 70 performs forward Fourier transform processing on the complex amplitude image to obtain a low-resolution image. The Fourier transform module 70 performs Fourier transform on the complex amplitude distribution in the complex amplitude image and outputs a low-resolution image containing the spectrum.

[0062] The frequency shift module 80 performs frequency shift processing on the low-resolution image to obtain a light display image. The frequency shift module 80 multiplies the spectrum in the low-resolution image with the pupil function and outputs multiple light display images containing sub-aperture spectra.

[0063] The inverse Fourier transform module 90 performs inverse Fourier transform processing on the light display image to obtain a high-resolution image. The inverse Fourier transform module 90 performs inverse Fourier transform on the sub-aperture spectra in the light display image and outputs a predicted high-resolution image.

[0064] Specifically, according to the forward process of Fourier ptychography, the process of irradiating M LED samples can be expressed as follows:

[0065]

[0066] where I p represents the light intensity captured by the sensor 20 under the simultaneous illumination of M lights, r represents the coordinates in the sample space, and I m represents the light intensity captured by the sensor 20 under the illumination of a single light, represents the inverse Fourier transform, O(k - k m ) represents the shift of the spectrum after the sample is illuminated by the tilted light, and P(k) represents the low-pass filtered pupil function.

[0067] The pupil function is:

[0068]

[0069] where NA is the numerical aperture of the objective lens, λ is the light wavelength, (k x , k y ) is the two-dimensional frequency domain coordinate system, and can be expressed by the formula k x= x / s for specific calculation, where s is the actual side length of the sample and x is the pixel distance of each point from the center.

[0070] The update module 40 calculates the loss function based on the initial image and the high-definition image.

[0071] Specifically, the update module 40 uses the L1 norm loss function to obtain a relatively stable gradient. According to the Fourier ptychography forward process combined with multi-lamp multiplexing, the loss function (loss) is:

[0072]

[0073] where I j represents the amplitude of the initially acquired image, and I p represents the amplitude of the high-definition image predicted by the network. Obtained by combining the formula satisfied by the sample in the forward process of Fourier ptychography

[0074]

[0075] In particular, to minimize the loss function, the network selects the Adam gradient descent algorithm as the optimizer.

[0076] The inspection module 100 continuously inspects the high-definition image and the number of updates. When the set standard is met, it outputs the reconstructed image. The inspection module 100 continuously inspects the resolution of the high-definition image and the number of updates of the update module 40 until the output image reaches the expected resolution or the number of updates reaches the custom number. The expected resolution and the number of updates standard can be set by the user according to actual needs. For example, the custom number of updates is usually between 20,000 and 50,000 times, which can reduce the computational amount and time cost required for updates while ensuring the quality of the reconstructed image.

[0077] To verify the outstanding effect of the method for implementing multiplexed Fourier ptychography imaging based on a priori neural network of the present invention, different datasets are used. Please refer to Figure 3 , and compare the phases of the images reconstructed by the method for implementing multiplexed Fourier ptychography imaging based on a priori neural network of the present invention and the existing Fourier ptychography imaging reconstruction methods; please refer to Figure 4 . After the method for implementing multiplexed Fourier ptychography imaging based on a priori neural network of the present invention and the existing Fourier ptychography imaging reconstruction methods respectively adopt the single-lamp algorithm and multi-lamp multiplexing, then compare the phases of the reconstructed images. It can be seen that regardless of which dataset and which dataset acquisition method are used, the contrast of the phases of the images reconstructed by the method for implementing multiplexed Fourier ptychography imaging based on a priori neural network of the present invention is higher than that of the existing Fourier ptychography imaging reconstruction methods, and the quality of the reconstructed images is good.

[0078] During the training process of the multiplexed Fourier ptychography imaging device based on the prior neural network of the present invention, for the training efficiency of the network, a batch is randomly selected from the sequence (1, 2, 3, …, N) for training. For each prediction map in this batch, the forward process of Fourier ptychography needs to be simulated successively according to the positions of M lights, and the sum of the squares of the amplitudes is obtained after accumulation. Compared with the existing Fourier ptychography of the single-light algorithm, the multi-light multiplexing utilized by the present invention saves a large amount of acquisition time and greatly improves the acquisition efficiency.

[0079] The above-described embodiments only represent the optimal implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and the present invention also intends to include these modifications and deformations.

Claims

1. A method for multiplexed Fourier stack imaging based on a priori neural network, characterized in that: include: An initial graph is obtained from the dataset; Restoring the phase of the initial image to obtain a phase restoration image; Restoring the amplitude of the initial image to obtain an amplitude restoration image; The phase recovery image and the amplitude recovery image are combined to obtain a complex amplitude image; Perform forward Fourier transform on the complex amplitude image to obtain a low-definition image; The low-definition image is subjected to frequency shift processing to obtain an optical display image; The optical display image is processed by inverse Fourier transform to obtain a high-definition image.

2. The method for realizing multiplexed Fourier stack imaging based on a priori neural network according to claim 1, characterized in that: The pupil function used in the frequency shift processing is: Where NA is the numerical aperture, λ is the wavelength of light, (k x ,k y ) is a two-dimensional frequency domain coordinate system.

3. The method for realizing multiplexed Fourier stack imaging based on a priori neural network according to claim 2, characterized in that: Also includes: The data set is obtained by using multiple lights multiplexing method. Multiple pictures are taken by changing different lighting directions and quantities to obtain the data set. The Fourier forward process of the sample after multi-lamp multiplexing acquisition satisfies: Among them, I p represents the light intensity captured under multiple illuminations at the same time, r represents the coordinate in the sample space, I m represents the light intensity captured under a single illumination, represents the inverse Fourier transform, O(kk m ) represents the shift in the spectrum when the sample is illuminated by oblique light.

4. The method for realizing multiplexed Fourier stack imaging based on a priori neural network according to claim 3, characterized in that: Also includes: Calculate the loss function based on the initial image and the high-definition image to update the restoration processing parameters; The loss function is: Among them, I j is the amplitude of the initial image collected, I p is the amplitude of the high-definition image predicted by the network, combined with the formula satisfied by the sample in the forward process of the Fourier stack, 5. The method for realizing multiplexed Fourier stack imaging based on a priori neural network according to claim 4, characterized in that: Also includes: The high-definition image and the number of updates are continuously checked, and the reconstructed image is output when the set standards are met.

6. A multiplexed Fourier stack imaging device based on a priori neural network, characterized in that: include: The acquisition module is used to acquire the initial graph from the data set; A phase restoration module, used to restore the phase of the initial image to obtain a phase restoration image; An amplitude restoration module is used to restore the amplitude of the initial image to obtain an amplitude restoration image; A synthesis module, used for synthesizing the phase restoration image and the amplitude restoration image to obtain a complex amplitude image; A Fourier transform module is used to perform forward Fourier transform processing on the complex amplitude image to obtain a low-definition image; A frequency shift module is used to perform frequency shift processing on the low-definition image to obtain an optical display image; The inverse Fourier transform module is used to perform inverse Fourier transform processing on the optical display image to obtain a high-definition image.

7. The device for realizing multiplexed Fourier stack imaging based on a priori neural network according to claim 6, characterized in that: The pupil function used in the frequency shift module is: Where NA is the numerical aperture, λ is the wavelength of light, (k x ,k y ) is a two-dimensional frequency domain coordinate system.

8. The device for realizing multiplexed Fourier stack imaging based on a priori neural network according to claim 7, characterized in that: It also includes a light source and a sensor, and uses a multi-light multiplexing method to obtain a data set for the sample; changing the different lighting directions and quantities of the light source, the sensor takes pictures to obtain multiple pictures, so as to obtain a data set; The Fourier forward process of the sample after multi-lamp multiplexing acquisition satisfies: Among them, I p represents the light intensity captured by the sensor under the simultaneous illumination of multiple light sources, r represents the coordinate in the sample space, I m It represents the light intensity captured by the sensor under the illumination of a single light source. represents the inverse Fourier transform, O(kk m ) represents the shift in the spectrum when the sample is illuminated by oblique light.

9. The device for realizing multiplexed Fourier stack imaging based on a priori neural network according to claim 8, characterized in that: It also includes an updating module, which is used to calculate a loss function based on the initial image and the high-definition image to update the restoration processing parameters; The loss function is: Among them, I j is the amplitude of the initial image collected, I p is the amplitude of the high-definition image predicted by the network, combined with the formula satisfied by the sample in the forward process of the Fourier stack, 10. The device for realizing multiplexed Fourier stack imaging based on a priori neural network according to claim 9, characterized in that: It also includes a testing module for continuously testing the high-definition image and the number of updates, and outputting a reconstructed image when the set standards are met.

Images