Anti-diffraction super-resolution lens-free imaging method based on random coding mask

Abstract

The invention discloses an anti-diffraction super-resolution lensless imaging method based on a random coding mask, belongs to the technical field of computational imaging, and aims to solve the problems in the prior art. Obtaining a point spread function (PSF) of the lens-free imaging system on the image sensor; establishing a general regularization equation according to the collected aliasing image and a calibrated point spread function (PSF); splitting the regularization equation; adding an activation function into the split regularization equation to obtain an improved regularization equation; solving the regularization equation to obtain an unknown number analytic expression and a hyper-parameter analytic expression; substituting the Fourier expression into the unknown number analytic expression and the hyper-parameter analytic expression in an operator form to obtain the unknown number analytic expression and the hyper-parameter analytic expression in the Fourier form; and substituting an aliasing image directly acquired by the lensless imaging system and a point spread function (PSF) of the system into the unknown number analytic expression and the hyper-parameter analytic expression, and iteratively calculating each unknown number and hyper-parameter until convergence.

Description

Technical Field

[0001] The invention belongs to the technical field of computational imaging, and in particular relates to an anti-diffraction super-resolution lensless imaging method based on a random coding mask. Background Art

[0002] The lensless imaging method is a novel computational imaging method, which usually uses light modulation elements such as coded masks to replace the lenses in traditional optical systems, that is, the target light field is coded and modulated directly before the image sensor, and the image of the scene is restored through computational reconstruction. Compared with traditional optical systems, lensless imaging systems greatly reduce the thickness, weight and moment of inertia of the system, and have the potential to achieve ultra-thin, ultra-light and low-cost; have negligible chromatic aberration, distortion and field curvature; have unlimited depth of field and the potential for passive three-dimensional imaging at close and medium distances; these advantages make lensless imaging systems have great application prospects in some fields with strict restrictions on volume and weight (such as micro unmanned platforms).

[0003] After the light of the target scene is modulated by the coding mask in the lensless imaging system, a corresponding projection will be generated on the image sensor. Although the projections of different incident light rays will always overlap densely to form an aliased image. However, since the coding mask is specially designed, even if the naked eye cannot recognize it, the overlapping light intensity distribution can be demodulated by a decoding method running on a computer to restore the light intensity information and reconstruct the target scene. Of course, due to the fewer physical restrictions on the light intensity distribution, the imaging results of the lensless imaging system are more susceptible to external and internal slight disturbances. This property is called ill-posed in mathematical physics, resulting in low spatial resolution and low image resolution. At the same time, there is currently no relevant research and discussion on the diffraction problem faced by the lensless imaging system.

[0004] See an academic article titled “DiffuserCam: lensless single-exposure 3D imaging” published by N.Antipa et al. in the journal Optica in 2018. In the article, N.Antipa et al. proposed a lensless camera called DiffuserCam, and used the lensless camera to extract the depth information of the target. The article proposed the use of an iterative regularization algorithm, TVAL3, to weaken the ill-posedness of the lensless camera. Although this type of method has to some extent solved the problem of difficulty in decoding due to the ill-posedness of the lensless imaging system in principle, the lensless imaging system based on this algorithm still has the problems of low spatial resolution and lack of anti-diffraction capability. Summary of the invention

[0005] In order to solve the problems of low spatial resolution and lack of anti-diffraction capability of existing lensless imaging systems, the present invention proposes an anti-diffraction super-resolution lensless imaging method based on a random coded mask, which can realize anti-diffraction super-resolution fast imaging of the lensless imaging system.

[0006] The technical solution of the present invention to solve the technical problem is:

[0007] A diffraction-resistant super-resolution lensless imaging method based on a random coding mask comprises the following steps:

[0008] Step 1, using an LED point light source to illuminate the lensless imaging system;

[0009] Step 2: Obtain the point spread function (PSF) of the lensless imaging system on the image sensor.

[0010] Step 3, establishing a general regularization equation based on the aliased image directly collected by the lensless imaging system and the calibrated point spread function PSF, and then running decoding on a computer;

[0011] Step 4, splitting the regularization equation in step 3 into two parts to speed up the convergence speed and convergence accuracy of the decoded image;

[0012] Step 5, adding an activation function to the regularization equation split in step 4 to suppress the inherent nonlinear characteristics of the lensless imaging system, thereby obtaining an improved regularization equation;

[0013] Step 6, solve the regularization equation in step 5 to obtain analytical expressions for unknowns and hyperparameters;

[0014] Step 7, substituting the Fourier expression into the unknown number analytical expression and the hyperparameter analytical expression of step 6 in the form of an operator to obtain the unknown number analytical expression and the hyperparameter analytical expression in Fourier form;

[0015] In step 8, the aliased image directly collected by the lensless imaging system and the point spread function PSF of the system are substituted into the analytical expressions of unknowns and hyperparameters in step 7, and each unknown and hyperparameter is calculated iteratively until convergence.

[0016] Beneficial effects of the present invention: The present invention combines the Fourier algorithm with the improved regularization algorithm in the form of an operator, forming a lensless imaging system decoding method with a conjugate structure, fast Fourier transform characteristics, and a simple structure. The method is essentially still a regularization algorithm, so it can still suppress the inherent ill-posedness of the lensless imaging system. At the same time, due to the simple structure and fast Fourier transform characteristics, it can have a faster calculation speed. Furthermore, since the method has a conjugate structure in design, it can suppress diffraction to achieve a super-resolution effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the principle of the present invention;

[0018] Figure 2 This is a schematic diagram of the random coding mask of the present invention;

[0019] Figure 3 This is a diagram of the point spread function PSF calibration result of the present invention;

[0020] Figure 4 This is a flow chart of the decoding method of the present invention;

[0021] Figure 5 This is an imaging effect diagram of the present invention;

[0022] Figure 6 It is the target scene test picture of the present invention;

[0023] Figure 7 A comparison diagram of the implementation of the method of the present invention;

[0024] Figure 1 In the figure, 1 is the target scene, 2 is the random coding mask, 3 is the image sensor, 4 is the aliased image, 5 is the computer, and 6 is the reconstructed target scene. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below with reference to the accompanying drawings.

[0026] like Figure 1 As shown, the system used in the anti-diffraction super-resolution lensless imaging method based on random coding mask includes three parts: random coding mask 2, image sensor 3 and computer 5. After the light of the target scene 1 is modulated by the random coding mask 2 in the lensless imaging system, a corresponding projection will be generated on the image sensor 3. The projections of different incident light rays will always overlap densely to form an aliased image 4. The random coding mask 2 is specially designed, and even if the naked eye cannot recognize it, the overlapping light intensity distribution can be demodulated by the decoding method running on the computer 5 to restore the light intensity information and reconstruct the target scene 6.

[0027] The random coding mask 2 uses a chrome-plated glass plate, the chrome-plated area is light-proof, and the non-chrome-plated area is light-transmissive. The size of the modulation unit on the random coding mask 2 is 8×8 microns. Figure 2 As shown, the random coding mask is a 0-1 binary matrix area: the 0 area is not transparent, Figure 2 The opaque modulation unit in the middle is represented by a black block; the area 1 is transparent, Figure 2 The light transmission modulation unit is represented by a white block. Image sensor 3 uses 1 / 1.8" Sony IMX334 CMOS, with a resolution of 3840×2160 and a size of each pixel of 2×2 microns. The CPU of computer 5 uses an 11th generation Intel i9 processor.

[0028] A diffraction-resistant super-resolution lensless imaging method based on a random coding mask comprises the following steps:

[0029] Step 1: Use an LED point light source to illuminate the lensless imaging system.

[0030] Calibrate the point spread function (PSF) of the lensless imaging system, which is a necessary input value for the computer to run the decoding of the anti-diffraction super-resolution lensless imaging method based on the random coding mask. Use the GCI-060411 white light LED light source produced by Daheng Optoelectronics to illuminate the lensless imaging system, align the LED light source and the lensless imaging system, and the distance between the two is 320mm.

[0031] Step 2: Get the point spread function (PSF) of the lensless imaging system on the image sensor. The result is as follows: Figure 3 shown.

[0032] Step 3, a general regularization equation is established based on the aliased image directly obtained by the lensless imaging system and the point spread function PSF of the system. The decoding method run by the computer is as follows Figure 4 shown.

[0033] First, establish the relationship between the aliased image I and the target scene O.

[0034] I=Φ·O (1)

[0035] Where Φ is the optically defined transfer function, which is the product expression of the point spread function PSF of the lensless imaging system. The symbol · in formula (1) represents the product of the matrix. Here, Φ is only an intermediate variable for establishing the regularization equation, which will be completely replaced by the point spread function PSF in the future, without the need for measurement and calculation. According to the relationship between the aliased image I and the target scene O in formula (1), the corresponding regularization equation is established.

[0036]

[0037] Step 4, split the regularization equation in step 3, and divide the original regularization expression into two parts to accelerate the convergence speed and convergence accuracy of the decoded image.

[0038] The formula factor (Φ·OI) in formula (2) is split into two parts, namely (Ix) and (Φ·Ox), to speed up the calculation speed and accuracy. Among them, x is the intermediate variable connecting the two split parts. Further, the formula is transformed into,

[0039]

[0040] Among them, ζ and ξ are hyperparameters in the regularization equation, and μ1 and μ2 are two custom coefficients used to adjust the convergence accuracy.

[0041] Step 5, add an activation function to the regularization equation split in step 4 to suppress the inherent nonlinear characteristics of the lensless imaging system, and obtain an improved regularization equation:

[0042] Add an activation function sgn(w), so the final regularization equation expression is,

[0043]

[0044] Step 6, solve the regularization equation in step 5 to obtain the analytical expressions for the unknowns and hyperparameters:

[0045] The unknowns in formula (4) are x, O and w. Correspondingly, the solution is:

[0046]

[0047] 1 represents a full 1 matrix in mathematics, the superscript H represents the conjugate transpose operation of the matrix, and the solutions of the three hyperparameters ζ, ξ, and ρ are,

[0048]

[0049] Step 7, substitute the Fourier expression into the unknown number analytical expression and hyperparameter analytical expression of step 6 in the form of an operator to obtain the unknown number analytical expression and hyperparameter analytical expression in Fourier form:

[0050] The operator form of the Fourier expression is,

[0051]

[0052] In formula (7) represents the fast Fourier transform, represents the conjugate operation of Fourier transform, and PSF is the point spread function of the lensless imaging system. Substituting it into formula (5) and formula (6),

[0053]

[0054]

[0055] Step 8: Substitute the aliased image I and the point spread function PSF directly collected by the lensless imaging system into the analytical expressions of unknowns and hyperparameters in step 7, and iteratively calculate each unknown and hyperparameter until convergence:

[0056] Substitute the aliased image I and the point spread function PSF directly collected by the lensless imaging system into formula (8) and formula (9), and iterate repeatedly until convergence. The convergence criterion is that the decoded image matrix no longer changes.

[0057] Generally, after the object is modulated by the point spread function PSF, an aliased image I can be directly obtained on the image sensor. Substitute the aliased image I and the point spread function PSF into formula (8) and formula (9), and take μ1 = 1 × 10 -6 , μ2=1×10 -6 , μ3=4×10 -5 , iterate repeatedly until convergence. The decoding effect is as follows Figure 5 shown. Figure 5 (a) is the actual working scenario of the camera. Figure 5 (b) is the imaging result of the red toolbox in the real scene. Figure 5 (c) to Figure 5 (i) shows the test results of the lensless camera imaging objects of yellow, green, blue, brown, white, blue-white mixture, and red-yellow mixture.

[0058] According to the diffraction formula,

[0059]

[0060] λ is the central wavelength of the camera, which is generally 0.55μm for visible light, and d is the distance from the encoding mask to the detector, which is 1.5mm here. It is concluded that the minimum aperture size within the diffraction range is 32μm, which is generally called the diffraction limit in optics. Since this algorithm has a conjugate structure and is very symmetrical in structure, it has the effect of eliminating diffraction.

[0061] In fact, after substituting the x expression in formula (8) into the restored target scene O expression, the following approximation can be made:

[0062]

[0063] in, The expression is represents the diffraction factor in the actual point spread function PSF, t is the transfer function of the aperture on the coding plate, represents the intensity distribution of diffracted light. It is a constant factor, which can be easily eliminated by the activation function sgn(w) introduced earlier, thus achieving the effect of anti-diffraction and super-resolution.

[0064] The comparison of the implementation of the method of the present invention is as follows:

[0065] Using the lensless imaging system of the present invention Figure 6 The target scene image shown is formed. Figure 6 The image is displayed on a monitor, and the distance between the monitor and the lensless camera is 320mm. Figure 7 As shown, for comparison, TVAL3 and gradient descent are also shown in Figure 7 superior.

[0066] The core of the quality level of lensless cameras is the ability to restore image features, so we introduce the "Scale Invariant Feature Transformation Algorithm" (SIFT) to measure the quality of lensless camera decoding. SIFT is a type of feature point extraction algorithm used to detect and describe local features in images. It finds extreme points in the spatial scale and extracts their position, scale, and rotation invariants. The essence of the SIFT algorithm is to find key points (feature points) in different scale spaces and calculate the direction of the key points. The key points found by SIFT are some very prominent points that will not change due to factors such as lighting, affine transformation and noise, such as corner points, edge points, bright spots in dark areas and dark spots in bright areas.

[0067] By comparing the decoding results of the resolution plate by the lensless camera decoding method of the present invention with those by TVAL3 and the gradient descent method, it can be concluded that the resolution and decoding speed of the present invention are ≤10.9 milliradians and 37.0 seconds, respectively, which are better than TVAL3 (≤21.9 milliradians, 47.3 seconds) and the gradient descent method (≤10.9 milliradians, 469.5 seconds). When evaluating the ability to restore image features by the SIFT algorithm, there are two main evaluation indicators, the number of matching feature points and the SIFT score. It can be seen that the anti-diffraction super-resolution decoding method of the lensless imaging system used in the present invention is better than TVAL3 and the gradient descent method in terms of both the number of matching feature points and the SIFT score. This shows that the anti-diffraction super-resolution lensless imaging system based on the random matrix coding template of the present invention has better image restoration capabilities.

[0068] In the above embodiment, the present invention and the anti-diffraction super-resolution decoding method used in the present invention can restore the object image with high resolution. Compared with using TVAL3 and gradient descent method under the same conditions, the imaging resolution is higher, the decoding speed is faster, and the image quality is better.

Claims

1. A diffraction-resistant super-resolution lensless imaging method based on a random coded mask, comprising the following steps: Step 1, using an LED point light source to illuminate the lensless imaging system; Step 2, obtaining the point spread function PSF of the lensless imaging system on the image sensor; Step 3, establishing a general regularization equation based on the aliased image directly collected by the lensless imaging system and the calibrated point spread function PSF, and then running decoding on a computer; Step 4, splitting the regularization equation in step 3 into two parts to speed up the convergence speed and convergence accuracy of the decoded image; Step 5, adding an activation function to the regularization equation split in step 4 to suppress the inherent nonlinear characteristics of the lensless imaging system, thereby obtaining an improved regularization equation; Step 6, solve the regularization equation in step 5 to obtain analytical expressions for unknowns and hyperparameters; Step 7, substituting the Fourier expression into the unknown number analytical expression and the hyperparameter analytical expression of step 6 in the form of an operator to obtain the unknown number analytical expression and the hyperparameter analytical expression in Fourier form; In step 8, the aliased image directly collected by the lensless imaging system and the point spread function PSF of the system are substituted into the analytical expressions of unknowns and hyperparameters in step 7, and each unknown and hyperparameter is calculated iteratively until convergence.

2. The method for anti-diffraction super-resolution lensless imaging based on a random coded mask according to claim 1, characterized in that: The decoding method of step 3 is performed by the computer: First, establish the relationship between the aliased image I and the target scene O. I=Φ·O (1) Wherein, Φ is the optically defined transfer function, which is the product expression of the point spread function PSF of the lensless imaging system. The symbol · in formula (1) represents the product of matrices. Here, Φ is only an intermediate variable for establishing the regularization equation, which will be completely replaced by the point spread function PSF in the future, without the need for measurement and calculation. According to the relationship between the aliased image I and the target scene O in formula (1), the corresponding regularization equation is established:

3. The method for anti-diffraction super-resolution lensless imaging based on random coded mask according to claim 1, characterized in that: The step 4 is specifically as follows: split the formula factor (Φ·OI) in formula (2) into two parts, namely, two factors (Ix) and (Φ·Ox), so as to speed up the calculation speed and calculation accuracy; wherein x is an intermediate variable connecting the two split parts. Further, the formula is transformed into: Among them, ζ and ξ are hyperparameters in the regularization equation, and μ1 and μ2 are two custom coefficients used to adjust the convergence accuracy.

4. The method for anti-diffraction super-resolution lensless imaging based on random coded mask according to claim 1, characterized in that: The step 5 is specifically as follows: adding an activation function sgn(w), so the final regularization equation expression is, 5. The method for anti-diffraction super-resolution lensless imaging based on random coded mask according to claim 1, characterized in that: The specific step 6 is: the unknowns in formula (4) are x, O and w, and accordingly, the solution can be obtained: 1 represents a full 1 matrix in mathematics, the superscript H represents the conjugate transpose operation of the matrix, and the solutions of the three hyperparameters ζ, ξ, and ρ are, 6. The anti-diffraction super-resolution lensless imaging method based on random coded mask according to claim 1, characterized in that: The specific step 7 is: the operator form of the Fourier expression is, In formula (7) represents the fast Fourier transform, represents the conjugate operation of Fourier transform, PSF is the point spread function of the lensless imaging system; Substituting it into formula (5) and formula (6), 7. The method for anti-diffraction super-resolution lensless imaging based on random coded mask according to claim 1, characterized in that: The step 8 is specifically as follows: Substitute the aliased image I and the point spread function PSF directly collected by the lensless imaging system into formula (8) and formula (9), and iterate repeatedly until convergence. The convergence criterion is that the decoded image matrix no longer changes.

Images