Lensless on-chip microscopic imaging system and method with sub-diffraction-limited translation of pixels

Abstract

The application discloses a kind of centimeter level translation's pixel super-resolution lensless on-chip microscopic imaging system and method, it is related to microscopic imaging technical field, system includes: light source, two transparent flat plate with multiple step structures, horizontal moving mechanism, object table, image detector and computer;Light beam emitted by light source is irradiated to the measured sample on the object table and object table after passing through two transparent flat plate;Image detector is used to detect holographic image after each movement of transparent flat plate;Computer is used to receive holographic image sequence, and high-resolution reconstruction image is obtained by applying pixel super-resolution fusion algorithm.The whole imaging system greatly simplifies traditional lensless on-chip holographic super-resolution imaging system, reduces the cost of whole imaging system, and the fusion algorithm of combination of convex set projection algorithm and alternate projection algorithm is handled sub-pixel displacement hologram, improves the precision of pixel super-resolution lensless on-chip holographic phase imaging.

Description

Technical Field

[0001] This invention relates to the field of microscopic imaging technology, and in particular to a centimeter-level translational pixel super-resolution lensless on-chip holographic microscopic imaging system and imaging method. Background Technology

[0002] In recent years, with the rapid development of computer technology and image sensor chip manufacturing processes, a series of novel computational microscopy imaging technologies have emerged. Lensless on-sheet holographic microscopy is a promising computational imaging technology. Lensless on-sheet holographic microscopes contain no optical lens elements; the sample is placed directly above the image sensor. It illuminates the sample with a coherent or partially coherent light source, and the direct-transmitted light field and the diffracted light field of the sample form a coaxial hologram on the image sensor surface. After the hologram is recorded, the light field of the sample is reconstructed using numerical calculations, ultimately allowing for the simultaneous acquisition of the sample's intensity and quantitative phase microscopic image. Compared to traditional microscopes, lensless on-sheet holographic microscopes can not only perform quantitative phase imaging on transparent samples but also simultaneously achieve a large field of view and high resolution, enabling high-resolution tracking observation of multiple biological samples within a large field of view. Furthermore, because lensless on-sheet holographic microscopes do not use traditional optical imaging systems, they offer advantages such as low cost, compact structure, and portability, thus holding promise as a next-generation POCT medical diagnostic tool.

[0003] However, since lensless on-plate holographic microscopes do not use an optical lens system, their imaging resolution is limited by the pixel size of the image sensor. Existing image sensors, such as CMOS sensors, have pixel sizes of approximately 1–2 μm. According to the sampling theorem, the limiting resolution of a lensless on-plate holographic microscope is twice the sensor pixel size. To overcome this limitation, pixel super-resolution techniques are commonly used to improve system resolution. Pixel super-resolution techniques can reconstruct high-resolution images from low-resolution image sequences with sub-pixel displacement. To obtain sub-pixel displacement image sequences, precise movement of the image sensor or sample is required, necessitating extremely high-precision mechanical displacement devices. Moving the light source and moving the sample or image sensor are equivalent. Although the system structure for moving the light source significantly reduces the precision requirements of the mechanical displacement platform, the system still requires a certain level of precision from the mechanical displacement device. Some researchers have proposed using fiber arrays to obtain sub-pixel displacement images without involving mechanical movement, but this requires precise alignment and positioning of fiber arrays with diameters on the micrometer scale, increasing both system complexity and cost. In addition, methods such as changing the wavelength, illumination angle, mask, rotating the camera plane, and rotating the parallel glass plate can be used to improve the system resolution. However, these methods either require high-precision tunable lasers or additional complex and precise mechanical devices such as precision rotating displacement stages, which increases the system cost and is not conducive to the application of lensless on-plate holographic microscopes.

[0004] On the other hand, in pixel-based super-resolution lensless on-sheet holographic microscopy based on subpixel displacement, the fusion algorithm of subpixel displacement holograms plays a crucial role in imaging resolution. Currently, commonly used pixel-based super-resolution fusion algorithms mainly include two categories: convex set projection fusion algorithm and alternating projection algorithm. The convex set projection algorithm only fuses the intensity distribution of the acquired holograms, without considering the special characteristics of holograms, which record not only intensity information but also phase information of the light wave field. Therefore, using the convex set projection algorithm to process subpixel displacement hologram fusion will cause deviations in the reconstructed phase, thus affecting the improvement of reconstruction resolution. Furthermore, this algorithm is very sensitive to initial values; the quality of the initial values ​​has a significant impact on the reconstruction results. Simultaneously, the solutions reconstructed by the convex set projection algorithm are not unique, and multiple solutions are unavoidable. Moreover, the iterative calculation of the convex set projection algorithm is very time-consuming, resulting in high computational costs. The alternating projection algorithm is similar to the GS phase retrieval algorithm, which involves repeated propagation projection between the object plane and the holographic plane, and fusing information from different holograms on the holographic plane to ultimately reconstruct high-resolution complex amplitude information. Because the alternating projection algorithm fully considers the characteristic of holograms simultaneously recording the amplitude and phase information of the object's light wave field, it is believed, in a sense, to better fuse holograms. However, in pixel super-resolution fusion, the fusion quality is greatly affected by the quality of the initial values, the accuracy of image registration, and image noise. Better initial value quality leads to better alternating projection reconstruction and can effectively reduce the number of iterations. As the number of iterations increases, image noise and image registration errors also have a greater impact on the reconstruction quality. Therefore, improving the quality of the initial values ​​can effectively reduce the number of iterations, reduce the impact of errors, and effectively improve the quality of pixel super-resolution reconstruction. Summary of the Invention

[0005] The purpose of this invention is to provide a pixel-level super-resolution lensless on-sheet microscopy imaging system and method with centimeter-level translation, which can reduce system cost and improve imaging accuracy.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] This invention provides a centimeter-level translational pixel super-resolution lensless on-sheet microscopy imaging system. The system includes: a light source, two orthogonally tilted transparent plates, a horizontal movement mechanism, a platform, an image detector, and a computer. Each transparent plate has multiple stepped structures, which are made of transparent material. The horizontal movement mechanism controls the horizontal movement of the two transparent plates.

[0008] The light beam emitted by the light source passes through the two transparent plates and then illuminates the platform and the sample to be tested on the platform. The image detector is used to detect the holographic image sequence formed by the light beam passing through the sample and output it to the computer. The holographic image sequence includes holographic images captured after each transparent plate is moved by a translation amount on the order of centimeters. When any transparent plate is moved by a translation amount equal to the length of a step, the displacement of the corresponding holographic image is a sub-pixel displacement. The length of a step is on the order of centimeters.

[0009] The computer is used to receive the holographic image sequence and apply a pixel super-resolution fusion algorithm to obtain a high-resolution reconstructed image; the pixel super-resolution fusion algorithm is a fusion algorithm combining the convex set projection algorithm and the alternating projection algorithm; the initial value of the alternating projection algorithm is obtained by single-frame phase recovery calculation of the high-resolution fused hologram obtained by the convex set projection algorithm fusion reconstruction.

[0010] The present invention also provides a method for centimeter-level translational pixel super-resolution lensless on-sheet holographic microscopy imaging, the method comprising:

[0011] Acquire the holographic image sequence captured by the image detector;

[0012] Image registration is performed on each holographic image in the holographic image sequence to obtain the displacement of each frame of the hologram;

[0013] A pixel super-resolution fusion algorithm is used to obtain a high-resolution reconstructed image for each of the registered holographic images; the pixel super-resolution fusion algorithm is a fusion algorithm combining the convex set projection algorithm and the alternating projection algorithm; the initial value of the alternating projection algorithm is obtained by single-frame phase recovery calculation from the high-resolution fused hologram obtained by the convex set projection algorithm fusion and reconstruction.

[0014] Optionally, image registration is performed on each hologram in the holographic image sequence to obtain the displacement of each frame of the hologram, specifically including:

[0015] The first holographic image is used as the reference image, and the other holographic images are used as the images to be registered.

[0016] Fourier transform is performed on the reference image and each of the images to be registered to obtain the spectrum of the reference image and the spectrum of the image to be registered.

[0017] Calculate the normalized cross power spectrum between the reference image and each of the images to be registered;

[0018] Each normalized cross-power spectrum is embedded into a zero matrix of N times the size to obtain a preprocessed cross-power spectrum, and an inverse Fourier transform is performed on the preprocessed cross-power spectrum to obtain a cross-correlation function.

[0019] Determine the peak position of the cross-correlation function, and determine the displacement amount between the reference image and each of the to-be-registered images according to the coordinates of the peak position.

[0020] Optionally, a pixel super-resolution fusion algorithm is used for each of the registered holographic images to obtain a high-resolution reconstructed image, which specifically includes:

[0021] Perform image fusion on each of the registered holographic images by using the projection onto convex sets algorithm to obtain the high-resolution fused hologram;

[0022] Apply a single-frame phase retrieval algorithm to the high-resolution fused hologram to obtain the initial object-plane high-resolution complex amplitude information;

[0023] Use the initial object-plane high-resolution complex amplitude information as the initial value in the alternating projection algorithm, and apply the alternating projection algorithm to each of the registered holographic images to obtain the high-resolution reconstructed image.

[0024] Optionally, perform image fusion on each of the registered holographic images by using the projection onto convex sets algorithm to obtain the high-resolution fused hologram, which specifically includes:

[0025] Upsample the first-frame holographic image by using bicubic interpolation to obtain an initial high-resolution image;

[0026] Simulate degradation on the m-th frame of the initial high-resolution image to obtain the m-th frame of the degraded image; m = 1, 2, 3,..., M; M is the total number of frames of the holographic images;

[0027] Compare the m-th frame of the degraded image with the actually detected m-th frame of the holographic image to obtain a residual value;

[0028] Correct the m-th frame of the initial high-resolution image according to the residual value to obtain the m-th frame of the corrected image; when m < M, the m-th frame of the corrected image is the (m + 1)-th frame of the initial high-resolution image;

[0029] Let m = m + 1, and return to the step of "simulating degradation on the m-th frame of the initial high-resolution image" until all the actually detected holographic images are traversed to obtain the M-th frame of the corrected image;

[0030] Judge whether the current fusion iteration number is equal to the first preset iteration number to obtain a first judgment result; each time the M-th frame of the corrected image is calculated, one fusion iteration is completed;

[0031] If the first judgment result is negative, then let m = 1, let the corrected image of the Mth frame be the initial high-resolution image of the 1st frame, and return to the step "simulate degradation of the initial high-resolution image of the mth frame";

[0032] If the first determination result is yes, then the corrected image in the current M-th frame is the high-resolution fused hologram.

[0033] Optionally, using the initial high-resolution complex amplitude information of the object plane as the initial value in the alternating projection algorithm, the alternating projection algorithm is applied to each registered holographic image to obtain the high-resolution reconstructed image, specifically including:

[0034] Using the initial high-resolution complex amplitude information of the object plane as the initial value in the alternating projection algorithm, the initial high-resolution complex amplitude information of the object plane is propagated to the holographic surface through the angular spectrum diffraction algorithm to obtain the complex amplitude information U of the holographic surface. m ;

[0035] For the complex amplitude information U m The intensity map I is obtained by taking the absolute value and squaring it. m ;

[0036] For the intensity map I m After blur downsampling, based on the displacement between the holographic image in the m-th frame and the holographic image in the first frame, the intensity map I after blur downsampling is... m Move to the hologram position in the m-th frame and then upsample to obtain intensity map I. r m m = 1, 2, 3….M; M is the total number of frames in the holographic image;

[0037] The intensity map The m-th upsampled holographic image is divided to obtain the division result;

[0038] The division result is then subjected to a reverse displacement between the displacement of the m-th frame holographic image and the first frame holographic image to obtain the division result after the reverse displacement.

[0039] Based on the division result after the reverse displacement and the complex amplitude information U m Obtain complex amplitude information U m' ;

[0040] According to the complex amplitude information U m and the complex amplitude information U m 'Update the complex amplitude information U' m The updated complex amplitude information is obtained.

[0041] The updated complex amplitude information is obtained using an angular spectrum diffraction algorithm. The complex amplitude distribution of the object plane is obtained by backpropagation back to the object plane. This complex amplitude distribution is then propagated to the holographic surface using an angular spectrum diffraction algorithm to obtain the complex amplitude information U of the holographic surface. m+1 ;

[0042] Let m = m + 1, then return to step "for the complex amplitude information U". m The intensity map I is obtained by taking the absolute value and squaring it. m ", until all frames of the holographic image have been traversed;

[0043] Determine if the current complete iteration count is the second preset iteration count; if the second determination result is no, then let m = 1, U 1 =U M+1 The complex amplitude of the holographic surface is cumulative, and the holographic symbol number starts from 1. Let the complex amplitude information U of the holographic surface be... M For the complex amplitude information U of the holographic surface 1 Return to step "for the complex amplitude information U" m The intensity map I is obtained by taking the absolute value and squaring it. m "; A complete iteration is completed when all frames of the holographic image are traversed."

[0044] If the first judgment result is yes, then the complex amplitude information of the object surface reconstructed at high resolution is obtained.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This invention provides a centimeter-level translational pixel-level super-resolution lensless on-sheet microscopy imaging system and method. Based on traditional lensless on-sheet microscopy, two transparent plates with stepped structures are orthogonally tilted between the light source and the sample under test. By horizontally moving these two transparent plates, sub-pixel displacement lensless on-sheet coaxial holograms can be acquired. The entire imaging system greatly simplifies traditional lensless super-resolution imaging systems, abandoning the original high-precision displacement stage (or high-precision rotation stage), reducing the cost of the entire imaging system. It can achieve micrometer-level displacement of the light source and sub-pixel displacement of the image under large step size and low precision hardware conditions. Furthermore, by processing the sub-pixel displacement hologram using a fusion algorithm combining convex set projection algorithm and alternating projection algorithm, the accuracy of pixel-level super-resolution lensless phase imaging can be improved. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a structural diagram of a centimeter-level translation pixel super-resolution lensless on-chip microscopy imaging system provided in Embodiment 1 of the present invention;

[0049] Figure 2 This is a schematic diagram of the stepped structure of the transparent flat plate provided in Embodiment 1 of the present invention;

[0050] Figure 3 This is a flowchart of a centimeter-level translation pixel super-resolution lensless on-chip holographic microscopy imaging method provided in Embodiment 2 of the present invention;

[0051] Figure 4 This is a graph showing the imaging performance test results of the pixel super-resolution lensless on-chip holographic microscopy system provided in Embodiment 2 of the present invention.

[0052] Figure 5 The reconstructed phase super-resolution imaging result provided in Embodiment 2 of the present invention.

[0053] Symbol explanation:

[0054] 1—Light source; 2—Transparent flat plate; 3—Object-carrying platform; 4—Image detector. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The purpose of this invention is to provide a centimeter-level translational pixel-level super-resolution lensless on-sheet microscopy imaging system and method. Based on traditional lensless on-sheet microscopy, this invention places two stepped glass plates orthogonally and obliquely between a light source and the sample under test. By horizontally moving these two stepped glass plates, a sub-pixel-shifted lensless on-sheet coaxial hologram can be acquired. Furthermore, a fusion algorithm combining convex set projection and alternating projection algorithms is used to process the sub-pixel-shifted hologram, ultimately obtaining higher-precision pixel-level super-resolution lensless phase imaging.

[0057] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment provides a centimeter-level translational pixel super-resolution lensless on-sheet microscopy imaging system. The system includes: a light source 1 (fiber-coupled LED illumination source), two orthogonally tilted transparent plates 2, a horizontal movement mechanism (including two low-precision, low-cost horizontal movement mechanisms), a platform 3, an image detector 4 (area array CMOS detector), and a computer; each transparent plate 2 has multiple stepped structures; the stepped structures are made of transparent material; the horizontal movement mechanism is used to control the horizontal movement of the two transparent plates 2.

[0060] The light beam emitted by the light source 1 passes through the two transparent plates 2 and then illuminates the object platform 3 and the sample to be tested on the object platform 3; the image detector 4 is used to detect the holographic sequence formed by the light beam passing through the sample and output it to the computer; the holographic image sequence includes holographic images taken after each transparent plate 2 is moved by a translation amount of one step length (cm); when any transparent plate 2 is moved by a translation amount of one step length (cm), the pixel displacement of the corresponding holographic image is a sub-pixel displacement.

[0061] Light source 1 is placed above the system as the system's illumination source 1. Light source 1 illuminates the sample to be tested placed on the sample carrier platform 3. After illuminating the sample, part of it is diffracted, and the other part is directly transmitted through the sample to continue propagating. The diffracted light and the transmitted light interfere on the receiving surface of the array detector to form an interference image, i.e., a coaxial hologram.

[0062] The computer is used to receive the holographic image sequence and apply a pixel super-resolution fusion algorithm to obtain a high-resolution reconstructed image; the pixel super-resolution fusion algorithm is a fusion algorithm combining the convex set projection algorithm and the alternating projection algorithm; the initial value of the alternating projection algorithm is obtained by calculating the phase recovery of a single-frame hologram through the high-resolution fused hologram reconstructed by the convex set projection algorithm.

[0063] In this setup, the light source 1 is 10-15 cm away from the platform 3, and the platform 3 is 0.5 mm-1 mm away from the image detector 4. Two transparent plates 2 are placed at an angle of approximately 10°-20° between the light source 1 and the platform 3. The glass steps can consist of 4-5 steps, and the displacement of the steps can be controlled by two horizontal displacement platforms to acquire 16-25 sub-pixel displacement holographic images.

[0064] The glass step structure is shown in the attached image. Figure 1 As shown. A special case of glass stepped structure, such as... Figure 2 As shown, the transparent plate 2 consists of 5 stacked biological slides forming 5 steps. Each biological slide is approximately 1 mm thick, and the slides are spaced 1.5 cm apart laterally. Figure 2 As shown. By horizontally moving the two orthogonal transparent plates 2, 25 subpixel displacement holograms can be acquired. Each transparent plate 2 has 5 steps. By fixing one transparent plate 2 and moving the other transparent plate 2, and moving one step at a time, a holographic image sequence in the form of a 5×5 matrix is ​​obtained.

[0065] like Figure 2 When light source 1 is incident at an angle onto a parallel glass layer of one thickness, the refracted light will be horizontally deflected. The horizontal distance of the deflection is related to the incident angle and the thickness of the parallel glass layer as shown in equation (1):

[0066]

[0067] Where l is the horizontal offset distance of the emitted light, θ is the incident angle, d is the thickness of the parallel glass layer, and n is the refractive index of the glass. When the incident angle is constant, the horizontal offset distance of the emitted light is proportional to the thickness of the parallel glass layer. By controlling the displacement platform to cause the glass step to move horizontally, light source 1 will sequentially be incident on parallel glass layers of different thicknesses at a constant angle, and the horizontal offset distance of the emitted light will also change due to the change in the thickness of the glass layer, thus realizing the equivalent micro-displacement of light source 1. A series of sub-pixel displacement coaxial holograms can be obtained.

[0068] In a lensless on-plate holographic microscope system, the method for acquiring subpixel displacement image sequences essentially involves moving the sample under test, moving the image detector 4, and moving the light source 1. Since the sample under test is very close to the image detector 4, the requirements for the mechanical movement devices for moving the sample and moving the image detector 4 are equivalent; their displacement amounts must be smaller than the pixel size of the CMOS sensor, meaning the displacement step size must reach the sub-micrometer level. Moving the light source 1 requires much lower precision. Based on the geometry of the lensless on-plate holographic microscope, the relationship between the displacement of the light source 1 and the displacement of the hologram is shown in equation (2):

[0069]

[0070] Where z2 is the distance between the sample being measured and the image detector 4, z1 is the distance between the light source 1 and the sample being measured, n1 is the refractive index of air, and n2 is the refractive index of the protective glass in front of the photosensitive unit of the CMOS sensor. If z2 = 0.75 mm, z1 = 15 cm, n1 = 1, and n2 = 1.5, then S hologram / S source ≈1 / 300, if the displacement of the captured hologram is 0.3μm, then the displacement of light source 1 is approximately 90μm. Therefore, the system structure for moving light source 1 significantly reduces the precision requirements of the mechanical displacement platform. However, despite this, the system still has certain precision requirements for the mechanical displacement device.

[0071] Since the refractive index of glass is greater than that of air, when light source 1 illuminates the glass plate at an angle, two refractions will occur. This refraction will cause light source 1 to undergo an equivalent lateral displacement, as shown in formula (1). When the incident angle is constant, the lateral displacement is only proportional to the thickness of the glass step and is independent of the length of the glass step. Therefore, by setting the length of the glass step, a large step size (the step size of the horizontal displacement of the glass step is determined by the length of the glass step) and a small displacement (the pixel displacement per frame is determined by the tilt angle of the glass step and the thickness of each step) can be achieved.

[0072] Compared to existing subpixel-shifted pixel super-resolution lensless on-chip systems, its significant advantages are: compared to existing schemes for acquiring subpixel-shifted images, this imaging system has almost no requirements for the precision of the subpixel mechanical displacement device (centimeter-level displacement), and can use inexpensive, low-precision mechanical displacement devices, which not only reduces the overall cost of the imaging system but also enhances its noise immunity. Figure 2 For example, assuming a step has a thickness of 1mm and a horizontal length of 1.48cm, and the glass has a refractive index of 1.5, when light source 1 is incident at a 10° angle with the step, translating one step is equivalent to causing a 59μm displacement of light source 1, thus resulting in a sub-pixel translation of the hologram. This 1.48cm displacement step can achieve sub-pixel displacement of the hologram, thus significantly reducing the precision requirements of the mechanical displacement platform. Furthermore, the angle at which the glass step is tilted, the length of the step, and the thickness of the step can be set according to actual needs, meaning the system parameters can be set very flexibly.

[0073] The above analysis shows that the system is highly robust to external noise. Since the lateral dimension of a step is on the order of centimeters, as long as the lateral offset of the displacement platform is not too large, it will not affect the offset of light source 1. If the tilt angle of the displacement platform changes slightly due to vibration, for example, a 5° deflection, changing the incident angle from 10° to 15°, this 5° deflection is already a relatively large error for the system. However, even if the displacement step size of light source 1 changes from 59 micrometers to 90 micrometers, it can still ensure that the image is shifted to sub-pixel level. Therefore, under normal circumstances, the impact on the offset of light source 1 is not significant, and the displacement of most images remains within one pixel. Therefore, the proposed sub-pixel displacement pixel super-resolution system not only has low requirements for the accuracy of the displacement platform but also has excellent resistance to external vibrations.

[0074] In this embodiment, two transparent flat plates 2 with glass step structures are orthogonally placed between the light source 1 and the object platform 3, which allows control of sub-pixel displacement of the image in two directions. The entire imaging system greatly simplifies the traditional lensless super-resolution imaging system, abandons the original high-precision displacement stage (or high-precision rotation stage), reduces the cost of the entire imaging system, and can achieve micron-level displacement of the light source 1 and sub-pixel displacement of the image under large step size and low precision hardware conditions.

[0075] Example 2

[0076] like Figure 3 As shown, this embodiment provides a pixel-level super-resolution holographic microscopy method based on the lensless on-sheet holographic microscopy system with centimeter-level translation provided in Embodiment 1. The method includes:

[0077] Q1: Obtain the holographic image sequence captured by image detector 4.

[0078] Partially coherent or coherent light is used as the light source 1. After passing through the transparent plate 2, the light source 1 is imaged on the image detector 4. For each step that the transparent plate 2 moves, the light source 1 is horizontally displaced by a distance. The horizontal moving mechanism makes the light source 1 tilt as it passes through parallel glass steps of different thicknesses. For each step that the light source 1 passes through, the image detector 4 acquires a subpixel displacement holographic image.

[0079] Q2: Perform image registration on each holographic image in the holographic image sequence to obtain the displacement of each frame of the hologram.

[0080] Subpixel displacement calculation of inter-frame holograms: Subpixel-level image registration algorithms are used to register the captured holograms, obtaining the displacement between each hologram. The subpixel-level phase correlation algorithm uses the Fast Fourier Transform (FFT) algorithm to transform the two images into the frequency domain, pads them with zeros in the frequency domain, calculates the cross-correlation between the two holograms, and identifies the position of the cross-correlation peak to determine the subpixel-level displacement between the two holograms.

[0081] Specifically, step Q2 includes:

[0082] Q21: Take the first frame of holographic image as the reference image f1(x,y), and take the other frames of holographic images as the images to be registered, denoted as f2(x,y).

[0083] Q22: Perform Fourier transform on the reference image f1(x,y) and each of the images to be registered f2(x,y) to obtain the spectrum F1(u,v) of the reference image and the spectrum F2(u,v) of the images to be registered.

[0084] Q23: Calculate the normalized cross power spectrum between the reference image and each of the images to be registered.

[0085] The normalized cross-power spectrum between the two images is:

[0086]

[0087] Q24: Embed each of the normalized cross-power spectra Q(u,v) into a zero matrix of N times the size to obtain the preprocessed cross-power spectra, and perform an inverse Fourier transform on the preprocessed cross-power spectra to obtain the cross-correlation function.

[0088] That is, embedding Q(u,v) into a zero matrix that is N times the size of Q(u,v) yields Q. up (u',v'), for example, if you want to get 1 / 20 pixel precision, you would embed Q(u,v) into a 0 matrix that is 20 times the size of Q(u,v), and then... up The inverse Fourier transform of (u', v') yields an upsampled cross-correlation function:

[0089] q up (x,y)=δ(x-Nx0,y-Ny0) (4)

[0090] Q25: Determine the cross-correlation function q up The peak position of (x,y) is used to determine the displacement between the reference image and each of the images to be registered, based on the coordinates of the peak position.

[0091] Find q upThe position of the peak (x, y) corresponds to a displacement of N times (Nx0, Ny0). Dividing this coordinate by N gives a displacement of 1 / N pixel precision (x0, y0).

[0092] Q3: A pixel super-resolution fusion algorithm is used to obtain a high-resolution reconstructed image for each of the registered holographic images; the pixel super-resolution fusion algorithm is a fusion algorithm combining the convex set projection algorithm and the alternating projection algorithm; the initial value of the alternating projection algorithm is obtained by single-frame phase recovery calculation of the high-resolution fused hologram obtained by the convex set projection algorithm fusion and reconstruction.

[0093] Step Q3 specifically includes:

[0094] Q31: The registered holographic images are fused using the convex set projection algorithm to obtain the high-resolution fused hologram.

[0095] Among them, the initial value of high resolution complex amplitude distribution is obtained by: (1) fusing the sequence of sub-pixel displacement holographic images obtained by the convex set projection algorithm (POCS) into a high resolution hologram; (2) using single-frame phase recovery technology to obtain the initial value of high resolution surface complex amplitude of the fused high resolution hologram.

[0096] Specifically, step 31 in Q includes:

[0097] Q311: Upsample the first frame of the holographic image using bicubic interpolation to obtain an initial high-resolution image S. 1 .

[0098] Q312: For the initial high-resolution image S of the m-th frame... m Simulated degradation is performed to obtain the degraded image L of the m-th frame. m m = 1, 2, 3..., M; M is the total number of frames in the holographic image.

[0099] The image degradation function for the m-th frame is given by where h is the degradation function:

[0100] L m =h×S m (5)

[0101] Q313: The degraded image L of the m-th frame... m The holographic image H obtained from the actual detection of the m-th frame m The comparisons are then performed to obtain the residual values.

[0102] Q314: Correct the initial high-resolution image S of the m-th frame based on the residual value. m, obtain the m-th corrected image; when m < M, the m-th corrected image is the (m + 1)-th initial high-resolution image S m+1 .

[0103] S m+1＝ S m + G(H m - L m )(6)

[0104] where G is a correction factor, and in the POCS algorithm, the correction factor is the projection operator

[0105] Q315: Let m = m + 1, and return to the step "simulate degradation of the m-th initial high-resolution image" until all the holographic images obtained from actual detections are traversed, and obtain the M-th corrected image S M+1 .

[0106] Q316: Determine whether the current fusion iteration count is equal to the first preset iteration count, and obtain the first judgment result; every time the M-th corrected image S is calculated M+1 then one fusion iteration is completed

[0107] If the first judgment result is no, then let m = 1, and let the M-th corrected image S M+1 be the first initial high-resolution image, and return to the step "simulate degradation of the m-th initial high-resolution image"

[0108] If the first judgment result is yes, then the current M-th corrected image S M+1 is the high-resolution fused hologram

[0109] Q32: Apply the single-frame phase retrieval algorithm to the high-resolution fused hologram to obtain the initial object-plane high-resolution complex amplitude information

[0110] Q33: Use the initial object-plane high-resolution complex amplitude information as the initial value in the alternating projection algorithm, and apply the alternating projection algorithm to each of the registered holographic images to obtain the high-resolution reconstructed image

[0111] The process involves iteratively updating the high-resolution complex amplitude distribution through alternating projection: First, the initial high-resolution object plane complex amplitude is propagated to the holographic plane using an angular spectrum propagation algorithm. High-resolution complex amplitudes are then obtained on the holographic plane, and their intensity is calculated to obtain high-resolution intensity information for the holographic plane. This high-resolution intensity information is then downsampled using a mean blurring algorithm and compared with the captured low-resolution holographic image. The comparison result is fed back to the high-resolution complex amplitude information, which is then corrected. The high-resolution complex amplitude is then propagated back to the object plane, remains unchanged, and then propagated to the holographic plane again. After downsampling on the holographic plane, it is compared with the next frame of the low-resolution hologram, and the high-resolution complex amplitude information is corrected. This process is repeated between the object plane and the holographic plane, fusing information from different frames on the holographic plane to ultimately obtain high-resolution intensity and phase images.

[0112] Specifically, step Q33 includes:

[0113] Q331: Using the initial high-resolution complex amplitude information of the object surface as the initial value in the alternating projection algorithm, the initial high-resolution complex amplitude information of the object surface is propagated to the holographic surface through the angular spectrum diffraction algorithm to obtain the complex amplitude information U of the holographic surface. m .

[0114] Q332: Regarding the complex amplitude information U m The intensity map I is obtained by taking the absolute value and squaring it. m .

[0115] That is, for complex amplitude U m Take strength I m =|U m | 2 .

[0116] Q333: Regarding the intensity map I m After blur downsampling, based on the displacement between the holographic image in the m-th frame and the holographic image in the first frame, the intensity map I after blur downsampling is... m Move to the position of the holographic image in the m-th frame, and then upsample to obtain the intensity map.

[0117] The intensity I m The image is blurred and downsampled, and then the intensity map I is generated based on the displacement (x0, y0) obtained from the registration in step Q2. m Corresponding hologram H m Registration, followed by upsampling to obtain

[0118] Q334: For the holographic image H in the m-th frame... m Upsampling is performed to obtain the m-th frame upsampled holographic image. The registered intensity map and the upsampled holographic image of the m-th frame Perform a division operation to obtain the division result C.

[0119] Q335: The result C of the division is reversed by the displacement obtained from image registration in step Q2 (-x0, -y0), and the result C of the division after the reverse displacement is obtained. r .

[0120] Q336: Based on the division result C after the reverse displacement r With the complex amplitude information U m Obtain complex amplitude information U m' Soon to be C r with U m Multiply to get U m' .

[0121] Q337: Based on the complex amplitude information U m and the complex amplitude information U m 'Update the complex amplitude information U' m The updated complex amplitude information is obtained.

[0122] Where α is the fusion coefficient, and its value is between 0 and 1.

[0123] Q338: Use the angular spectrum diffraction algorithm to update the complex amplitude information The complex amplitude distribution of the object plane is obtained by backpropagation back to the object plane. This complex amplitude distribution is then propagated to the holographic surface using an angular spectrum diffraction algorithm to obtain the complex amplitude information U of the holographic surface. m+1 The sequence number of the super-resolution image is not related to the frame number of the image; it is simply a result of calculation.

[0124] Q339: Let m = m + 1, return to step "for the complex amplitude information U m The intensity map I is obtained by taking the absolute value and squaring it. m ", until the holographic image has been traversed through all frames.

[0125] Q3310: Determine whether the current complete iteration count is the second preset iteration count; if the second determination result is no, then let m = 1, U 1 =U M+1 Among them, the holographic complex amplitude label is cumulative, and the holographic icon number starts from 1. Return to step "for the complex amplitude information U". m The intensity map I is obtained by taking the absolute value and squaring it. m ".

[0126] A complete iteration is completed when all subpixel displacements of the holographic image are traversed. When a complete iteration ends, the next iteration starts a new iteration from the first frame. The iteration terminates when the iteration termination condition is met.

[0127] If the first judgment result is yes, then the high-resolution complex amplitude information of the object surface is obtained.

[0128] Compared with existing algorithms, the method provided by this invention has the following significant advantages: 1) It can simultaneously achieve phase retrieval and pixel super-resolution, directly obtaining high-resolution complex amplitude information. 2) Based on the pixel super-resolution of the convex set projection algorithm, the alternating projection iteration of the object plane and image plane can further optimize phase information and improve resolution. 3) Using the convex set projection algorithm as the initial value can effectively reduce the number of alternating projection iterations, reduce the interference of image noise and registration errors on the alternating projection algorithm, and effectively improve the reconstruction quality of the alternating projection fusion algorithm. 4) Using a sub-pixel level phase correlation algorithm to obtain the displacement of each image relative to the reference standard image can quickly and accurately obtain the precise position of each image. This preprocessing can effectively improve the quality of pixel super-resolution and reduce the program running time.

[0129] To test the imaging performance of this pixel-level super-resolution lensless on-chip holographic microscopy system, the USAF1951 high-resolution board was first used for testing. In the constructed experimental system, the illumination source was a multimode fiber-coupled LED light source (M455F3, Thorlabs, center wavelength 455nm), and the detectors were four CMOS image detectors (DMM).

[0130] 27UJ003-ML (The Imaging Source) – This device has a pixel size of 1.67μm × 1.67μm, 10 million pixels, and a field of view of 29.85mm. 2 The distance z2 between the sensor plane and the sample plane is ~0.75mm, and the distance z1 between the illumination source and the sample plane is ~15cm. Two inclined glass plates with stepped structures are placed between the light source 1 and the sample plane. Each plate is composed of four glass slides (Shitai, 1mm thick, 75×25mm in size). 2 The hologram is formed by superimposing the light sources, with an angle of approximately 20° to the vertical direction. The step length is approximately 1.9 cm. According to the theoretical calculation based on formula (1), the displacement step of the light source 1 is approximately 121 μm for each step. According to formula (2), the hologram displacement step is 0.4 μm, and the pixel size of the CMOS sensor is 1.67 μm, therefore the hologram displacement step is approximately 0.24 pixels. The experimental results are as follows: Figure 4 As shown, where Figure 4(a) shows the recovery result after iterative reconstruction using the proposed pixel super-resolution method. Figure 4 (b) is Figure 4 (a) is a magnified view of a selected portion. Figure 4 (d) shows the result of direct phase retrieval of a single-frame hologram. Figure 4 (e) is Figure 4 (d) is a magnified view of a selected portion. Figure 4 (c) and (f) are respectively Figure 4 The cross-sections of corresponding colors in (b) and (e) are from... Figure 4 As can be seen from (f), the direct reconstruction of the full width at half maximum (FWHM) resolution can reach 1.74 μm, while from... Figure 4 (c) It can be seen that the half-width resolution can reach 0.87μm after reconstruction by the present invention. The pixel size of the CMOS sensor in the system is 1.67μm. Therefore, the method of the present invention can realize large field-of-view pixel super-resolution imaging, and the reconstructed image has a good signal-to-noise ratio.

[0131] Using viable yeast cells as samples (without any pretreatment such as staining), a small amount of yeast-containing culture medium was extracted from a culture dish and dropped onto a glass slide using a pipette. A coverslip was then placed on the slide to create a specimen, which was then placed on a stage for observation. The displacement platforms in the x and y directions were controlled to allow the illumination source to pass sequentially through parallel glass layers of different thicknesses, resulting in the acquisition of low-resolution holographic images of sub-pixel displacement. The reconstructed phase super-resolution imaging results are shown below. Figure 5 As shown, where Figure 5 (a) is a phase diagram of a yeast sample under a wide field of view. Figure 5 In (a), the rectangles b, c, and d represent the three yeast cells selected and reconstructed respectively. Figure 5 (b1)(c1)(d1) are the results obtained by direct phase recovery of the hologram. Figure 5 (b1), (c1), and (d1) correspond to the three yeast cells represented by rectangles b, c, and d, respectively. Figure 5 (b2)(c2)(d2) are the results of the recovery after iterative reconstruction using this method. Figure 5 (b2)(c2)(d2) correspond to the three yeast cells represented by rectangles b, c, and d, respectively. Figure 5 (b3)(c3)(d3) are respectively Figure 5 (b1)(c1)(d1) and Figure 5 The comparison diagrams of the cutoff lines in (b2), (c2), and (d2) show that, compared with the experimental results, the method of the present invention can clearly demonstrate that it can improve the resolution of phase imaging and the reconstructed image has a good signal-to-noise ratio.

[0132] Example 3

[0133] This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the centimeter-level translation pixel super-resolution lensless on-sheet holographic microscopy imaging method of Embodiment 1.

[0134] Alternatively, the aforementioned electronic device may be a server.

[0135] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the centimeter-level translational pixel super-resolution lensless on-sheet holographic microscopy imaging method of Embodiment 1.

[0136] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A centimeter-level translational pixel super-resolution lensless on-sheet microscopy imaging system, characterized in that, The system includes: a light source, two orthogonally tilted transparent plates, a horizontal moving mechanism, a loading platform, an image detector, and a computer; each transparent plate has multiple step structures; the step structures are made of transparent material; the horizontal moving mechanism is used to control the horizontal movement of the two transparent plates; The light beam emitted by the light source passes through the two transparent plates and then illuminates the platform and the sample to be tested on the platform. The image detector is used to detect the holographic image sequence formed by the light beam passing through the sample and output it to the computer. The holographic image sequence includes holographic images captured after each transparent plate is moved by a translation amount on the order of centimeters. When any transparent plate is moved by a translation amount equal to the length of a step, the displacement of the corresponding holographic image is a sub-pixel displacement. The length of a step is on the order of centimeters. The computer is used to receive the holographic image sequence and apply a pixel super-resolution fusion algorithm to obtain a high-resolution reconstructed image; the pixel super-resolution fusion algorithm is a fusion algorithm combining the convex set projection algorithm and the alternating projection algorithm; the initial value of the alternating projection algorithm is obtained by single-frame phase recovery calculation of the high-resolution fused hologram obtained by the convex set projection algorithm fusion reconstruction. The application of pixel super-resolution fusion algorithms to obtain high-resolution reconstructed images specifically includes: Image registration is performed on each holographic image in the holographic image sequence to obtain the displacement of each frame of the hologram; The registered holographic images are fused using the convex set projection algorithm to obtain the high-resolution fused hologram. The initial high-resolution complex amplitude information of the object surface is obtained by applying a single-frame phase recovery algorithm to the high-resolution fused hologram. Using the initial high-resolution complex amplitude information of the object surface as the initial value in the alternating projection algorithm, the alternating projection algorithm is applied to each of the registered holographic images to obtain the high-resolution reconstructed image.

2. A method for achieving pixel-level super-resolution holographic microscopy based on the lensless on-sheet holographic microscopy system with centimeter-level translation as described in claim 1, characterized in that, The method includes: Acquire the holographic image sequence captured by the image detector; Image registration is performed on each holographic image in the holographic image sequence to obtain the displacement of each frame of the hologram; A pixel super-resolution fusion algorithm is used to obtain a high-resolution reconstructed image from each of the registered holographic images; the pixel super-resolution fusion algorithm is a fusion algorithm combining the convex set projection algorithm and the alternating projection algorithm; the initial value of the alternating projection algorithm is obtained by single-frame phase recovery calculation from the high-resolution fused hologram obtained by the convex set projection algorithm fusion and reconstruction. Specifically, a pixel super-resolution fusion algorithm is used to obtain high-resolution reconstructed images from each of the registered holographic images, including: The registered holographic images are fused using the convex set projection algorithm to obtain the high-resolution fused hologram. The initial high-resolution complex amplitude information of the object surface is obtained by applying a single-frame phase recovery algorithm to the high-resolution fused hologram. Taking the initial object plane high-resolution complex amplitude information as the initial value in the alternating projection algorithm, applying the alternating projection algorithm to each of the registered holographic images to obtain the high-resolution reconstructed image.

3. The method according to claim 2, characterized in that, Performing image registration on each holographic image in the holographic image sequence to obtain the displacement amount of each frame of the hologram, specifically including: Taking the first frame of the holographic image as the reference image and each of the other frames of the holographic images as the images to be registered; Performing Fourier transforms on the reference image and each of the images to be registered respectively to obtain the spectrum of the reference image and the spectrum of the image to be registered; Calculating the normalized cross-power spectrum between the reference image and each of the images to be registered; Embedding each of the normalized cross-power spectra into a zero matrix of N times the size to obtain the preprocessed cross-power spectrum, and performing an inverse Fourier transform on the preprocessed cross-power spectrum to obtain the cross-correlation function; Determining the peak position of the cross-correlation function, and determining the displacement amount between the reference image and each of the images to be registered according to the coordinates of the peak position.

4. The method according to claim 2, characterized in that, Performing image fusion on each of the registered holographic images using the projection onto convex sets algorithm to obtain the high-resolution fused hologram, specifically including: Performing upsampling on the first frame of the holographic image using bicubic interpolation to obtain an initial high-resolution image; Simulating degradation on the m-th frame of the initial high-resolution image to obtain the m-th degraded image; m = 1, 2, 3,..., M; M is the total number of frames of the holographic images; Comparing the m-th degraded image with the actually detected m-th holographic image to obtain a residual value; Correcting the m-th frame of the initial high-resolution image according to the residual value to obtain the m-th corrected image; when m < M, the m-th corrected image is the (m + 1)-th frame of the initial high-resolution image; Letting m = m + 1, and returning to the step of "simulating degradation on the m-th frame of the initial high-resolution image" until all the actually detected holographic images are traversed to obtain the M-th corrected image; Judging whether the current fusion iteration number is equal to the first preset iteration number to obtain a first judgment result; each time the M-th corrected image is calculated, one fusion iteration is completed; If the first judgment result is no, then letting m = 1, letting the M-th corrected image be the first frame of the initial high-resolution image, and returning to the step of "simulating degradation on the m-th frame of the initial high-resolution image"; If the first judgment result is yes, then the current M-th corrected image is the high-resolution fused hologram.

5. The method according to claim 2, characterized in that, Taking the initial object plane high-resolution complex amplitude information as the initial value in the alternating projection algorithm, applying the alternating projection algorithm to each of the registered holographic images to obtain the high-resolution reconstructed image, specifically including: Using the initial high-resolution complex amplitude information of the object plane as the initial value in the alternating projection algorithm, the initial high-resolution complex amplitude information of the object plane is propagated to the holographic surface through an angular spectrum diffraction algorithm to obtain the complex amplitude information of the holographic surface. ; For the complex amplitude information The intensity map is obtained by taking the absolute value and squaring it. ; For the intensity map After blur downsampling, based on the displacement between the holographic image in the m-th frame and the holographic image in the first frame, the intensity map after blur downsampling is... Move to the position of the holographic image in the m-th frame, and then upsample to obtain the intensity map. m = 1, 2, 3….M; M is the total number of frames in the holographic image; The intensity map The m-th upsampled holographic image is divided to obtain the division result; Performing reverse displacement on the division result using the displacement amount between the m-th frame of the holographic image and the first frame of the holographic image to obtain the division result after reverse displacement; Based on the division result after the reverse displacement and the complex amplitude information Obtain complex amplitude information ; According to the complex amplitude information and the complex amplitude information Update the complex amplitude information The updated complex amplitude information is obtained. ; The updated complex amplitude information is obtained using an angular spectrum diffraction algorithm. The complex amplitude distribution of the object plane is obtained by backpropagation back to the object plane. This complex amplitude distribution is then propagated to the holographic surface using an angular spectrum diffraction algorithm to obtain the complex amplitude information of the holographic surface. ; Let m = m + 1, then return to step "for the complex amplitude information". The intensity map is obtained by taking the absolute value and squaring it. ", until all frames of the holographic image have been traversed; Determine if the current complete iteration count is the second preset iteration count to obtain the second determination result; if the second determination result is not, then set m=1, U 1 =U M+1 In this context, the holographic complex amplitude is cumulative, and the holographic icon number starts from 1. Returning to step "Regarding the complex amplitude information..." The intensity map is obtained by taking the absolute value and squaring it. "; A complete iteration is completed when all frames of the holographic image are traversed." If the first judgment result is yes, then obtaining the object plane high-resolution complex amplitude information.

Images