A three-dimensional reconstruction method, device and storage medium based on microlens array

Abstract

The present invention provides a three-dimensional reconstruction method, device, and storage medium based on a microlens array, belonging to the field of light field microscopy. The method comprises: importing multiple light field raw data, processing white images of the multiple light field raw data to obtain a target white image; screening the coordinates with the maximum brightness value in the target white image, obtaining multiple microlens center coordinates after screening, and constructing the microlens array surface coordinates through all the microlens center coordinates; obtaining a sub-aperture image through a CCD camera, importing point light source data, and analyzing the sub-aperture image, point light source data, and microlens array surface coordinates to obtain a three-dimensional reconstruction result. The present invention can obtain light field microscopy imaging results at different imaging depths, and the three-dimensional reconstruction effect is relatively good, which is suitable for three-dimensional reconstruction of light field microscopy.

Description

Technical Field

[0001] The present invention mainly relates to the field of light field microscopy technology, and in particular to a three-dimensional reconstruction method, device and storage medium based on a microlens array. Background Art

[0002] Light-field microscopy is a modification of conventional microscopes by introducing a microlens array into the optical path. This allows for simultaneous acquisition of spatial and angular data on the imaged object. Because light-field data is captured in a single shot, light-field microscopy enables scanning-free 3D imaging at the frame rate of conventional cameras. Its high-speed 3D imaging capabilities hold great potential for application in biomedicine.

[0003] Optical microscopes, as essential tools for observing the microscopic world, help humans observe microbial cells, promote the development of biological cytology, and greatly facilitate biological research. Light-field microscopy, a scanning-free 3D microscopic computational imaging system, simultaneously records both spatial and angular information of light, enabling high-speed 3D imaging and possessing great potential for various biological imaging applications. Currently, light-field microscopy has been applied to live cell imaging, zebrafish neuron imaging, cardiac imaging, and blood flow monitoring.

[0004] Due to the existence of diffraction effects, traditional three-dimensional reconstruction algorithms based on geometric optics find it difficult to obtain light field microscopy imaging results at different imaging depths, and the three-dimensional reconstruction effect is poor, so it is not suitable for three-dimensional reconstruction of light field microscopy. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and to provide a three-dimensional reconstruction method, device and storage medium based on a microlens array.

[0006] The present invention solves the above-mentioned technical problem with the following technical solution: A three-dimensional reconstruction method based on a microlens array comprises the following steps:

[0007] Importing a plurality of light field raw data, performing white image processing on the plurality of light field raw data to obtain a target white image;

[0008] Screening the coordinates with the largest brightness value in the target white image, obtaining a plurality of microlens center coordinates after screening, and constructing the microlens array surface coordinates through all the microlens center coordinates;

[0009] A sub-aperture image is obtained by a CCD camera, and point light source data is imported. The sub-aperture image, the point light source data and the microlens array surface coordinates are analyzed to obtain an estimated value, and the estimated value is used as a result of three-dimensional reconstruction.

[0010] Another technical solution of the present invention to solve the above technical problem is as follows: a three-dimensional reconstruction device based on a microlens array, comprising:

[0011] A white image processing module is used to import a plurality of light field raw data, perform white image processing on the plurality of light field raw data, and obtain a target white image;

[0012] An image screening module is used to screen the coordinates with the largest brightness value in the target white image, obtain multiple microlens center coordinates after screening, and construct the microlens array surface coordinates through all the microlens center coordinates;

[0013] The 3D reconstruction result acquisition module is used to obtain a sub-aperture image through a CCD camera, import point light source data, analyze the sub-aperture image, the point light source data and the microlens array surface coordinates to obtain an estimated value, and use the estimated value as the result of the 3D reconstruction.

[0014] Another technical solution of the present invention to solve the above-mentioned technical problem is as follows: A three-dimensional reconstruction device based on a microlens array includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the three-dimensional reconstruction method based on the microlens array as described above is implemented.

[0015] Another technical solution of the present invention to solve the above technical problem is as follows: a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the three-dimensional reconstruction method based on the microlens array as described above.

[0016] The beneficial effects of the present invention are as follows: a target white image is obtained by processing white images of a plurality of light field raw data, the coordinates with the largest brightness value in the target white image are screened, a plurality of microlens center coordinates are obtained after screening, and the microlens array surface coordinates are constructed by all the microlens center coordinates. The sub-aperture image, point light source data and microlens array surface coordinates are analyzed to obtain the result of three-dimensional reconstruction, and light field microscopy imaging results at different imaging depths can be obtained, and the three-dimensional reconstruction effect is relatively good, which can be applicable to the three-dimensional reconstruction of light field microscopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic flow chart of a three-dimensional reconstruction method based on a microlens array provided in an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a point light source light field microscopic imaging process of a three-dimensional reconstruction method based on a microlens array provided in an embodiment of the present invention;

[0019] Figure 3This is a module block diagram of a three-dimensional reconstruction device based on a microlens array provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0021] Figure 1 A schematic flow chart of a three-dimensional reconstruction method based on a microlens array provided in an embodiment of the present invention.

[0022] like Figure 1 As shown, a three-dimensional reconstruction method based on a microlens array includes the following steps:

[0023] Importing a plurality of light field raw data, performing white image processing on the plurality of light field raw data to obtain a target white image;

[0024] Screening the coordinates with the largest brightness value in the target white image, obtaining a plurality of microlens center coordinates after screening, and constructing the microlens array surface coordinates through all the microlens center coordinates;

[0025] A sub-aperture image is obtained by a CCD camera, and point light source data is imported. The sub-aperture image, the point light source data and the microlens array surface coordinates are analyzed to obtain an estimated value, and the estimated value is used as a result of three-dimensional reconstruction.

[0026] It should be understood that sub-aperture images are obtained for three-dimensional imaging of objects. Different microlens arrays record different angular information of the imaged object, providing a prerequisite for three-dimensional reconstruction of the object. The sub-aperture image is a collection of light from a certain part of the main lens. Although it is visually difficult to distinguish the difference between the images, sub-aperture images at different positions show different angular information.

[0027] Specifically, the process of obtaining a sub-aperture image using a CCD camera is as follows: based on the calibrated center of the white image, a square image area with a fixed pixel point can be extracted with the center point of the microlens image as the center. The extracted image is photographed according to the original relative position and the center point is marked. All pixels at the same position in the microlens image are taken out and rearranged to obtain the sub-aperture image of the desired viewpoint.

[0028] In the above embodiment, a target white image is obtained by processing white images of a plurality of raw light field data, and the coordinates with the largest brightness value in the target white image are screened. After screening, a plurality of microlens center coordinates are obtained, and the microlens array surface coordinates are constructed using all the microlens center coordinates. The sub-aperture image, point light source data, and microlens array surface coordinates are analyzed to obtain a three-dimensional reconstruction result. Light field microscopy imaging results at different imaging depths can be obtained, and the three-dimensional reconstruction effect is relatively good, which is suitable for three-dimensional reconstruction of light field microscopy.

[0029] Optionally, as an embodiment of the present invention, the process of performing white image processing on the plurality of raw light field data to obtain a target white image includes:

[0030] extracting a plurality of original white images corresponding to each of the light field original data respectively from each of the light field original data;

[0031] Calibrate each of the extracted original white images to obtain a calibrated white image corresponding to each of the original white images;

[0032] Performing denoising processing on each of the calibrated white images to obtain a denoised white image corresponding to each of the original white images;

[0033] Perform mean processing on all denoised white images to obtain the target white image.

[0034] It should be understood that the white image (ie, the original white image) is calibrated to obtain a plurality of light field raw data and extract a plurality of white images (ie, the original white image).

[0035] Specifically, median filtering is used for denoising, and the denoised white images are averaged to generate a new white image (i.e., the target white image). Since the center of the lens has the strongest light focusing ability, the center energy coordinates of the basic white image are found.

[0036] It should be understood that the acquisition of the white image (ie, the target white image) is arranged before the entire light field imaging system is started, and is used to serve as a physical check of the light field system.

[0037] In the above embodiment, multiple original white images are extracted from each light field raw data, each original white image is calibrated to obtain a calibrated white image, each calibrated white image is denoised to obtain a denoised white image, and the average of all denoised white images is processed to obtain a target white image, which can serve as a physical inspection of the light field system and provide data support for subsequent data processing.

[0038] Optionally, as an embodiment of the present invention, the process of screening the coordinates with the maximum brightness value in the target white image and obtaining the coordinates of the multiple microlenses centers after screening includes:

[0039] The MATLAB tool is used to screen the coordinates with the maximum brightness value in the target white image, and after screening, multiple microlens center coordinates are obtained.

[0040] It should be understood that the MATLAB tool is a commercial mathematical software produced by MathWorks, an American company, and is used in data analysis, wireless communications, deep learning, image processing and computer vision, signal processing, quantitative finance and risk management, robotics, control systems and other fields.

[0041] Specifically, the coordinates of the center point of the white image are calculated for performing distortion inspection and correction processing on the lens after acquiring the white image (ie, the target white image), and a light field camera and a microlens are used for collection, digitization and correction.

[0042] In the above embodiment, MATLAB tools are used to screen the coordinates with the maximum brightness value in the target white image. After screening, multiple microlens center coordinates are obtained, which can be used to check and correct lens distortion, provide data support for subsequent data processing, and obtain light field microscopy imaging results at different imaging depths.

[0043] Optionally, as an embodiment of the present invention, the point light source data includes point light source coordinates and point light source intensity;

[0044] The process of analyzing the sub-aperture image, the point light source data, and the microlens array surface coordinates to obtain an estimated value includes:

[0045] Calculating the light intensity distribution of the point light source coordinates, the point light source intensity, and the microlens array surface coordinates based on the PSF model to obtain the light intensity distribution;

[0046] An estimated value is calculated for the light intensity distribution and the sub-aperture image to obtain an estimated value.

[0047] It should be understood that the light intensity distribution refers to the use of a curve or table to represent the luminous intensity values ​​of a light source or lamp in various directions in space.

[0048] It should be understood that the known PSF of the microscopic imaging system is a prerequisite for three-dimensional reconstruction of the object. Based on the knowledge of wave optics, the imaging of a point light source is simulated and the PSF (point spread function) of the microscopic imaging system can be derived according to the calculation formula.

[0049] In the above embodiment, the light intensity distribution is calculated based on the PSF model for the point light source coordinates, the point light source intensity and the microlens array surface coordinates, and the estimated values ​​of the light intensity distribution and the sub-aperture image are calculated to obtain estimated values. Light field microscopy imaging results at different imaging depths can be obtained, and the three-dimensional reconstruction effect is relatively good, which can be suitable for three-dimensional reconstruction of light field microscopy.

[0050] Optionally, as an embodiment of the present invention, the process of calculating the light intensity distribution based on the PSF model for the point light source coordinates, the point light source intensity, and the microlens array surface coordinates to obtain the light intensity distribution includes:

[0051] The light intensity distribution is calculated based on the PSF model for the coordinates of the point light source, the intensity of the point light source, and the coordinates of the microlens array surface to obtain the light intensity distribution. The PSF model is:

[0052]

[0053] in,

[0054] in,

[0055] Among them, α≈arcsin(NA / n),

[0056] Among them, ω u1 is the spatial frequency of the pixel with coordinate u1 on the CCD camera, ω v1 is the spatial frequency of the pixel at coordinate v1 on the sensor, and where h(x,y,z) is the light intensity distribution, U(x,y,z) is the spherical wave of the microlens array, T(x,y,z) is the microlens array, Φ(x,y,z) is the phase mask, D is the numerical aperture of the microlens, s is the abscissa of the microlens array surface, t is the ordinate of the microlens array surface, comb is the two-dimensional comb function, f0 is the focal length of the microscope objective, M is the magnification, λ is the intensity of the point light source, α is the maximum angle of incidence of the objective aperture, J0 is the Bessel function, rect() is the rectangular function, || ||2 is the Euclidean norm, (x,y,z) are the coordinates of the point light source, n is the refractive index of the lens, and NA is the numerical aperture of the objective.

[0057] It should be understood that the Euclidean norm, that is, the Euclid norm (commonly used to calculate the length of a vector), is the square root of the sum of the squares of the absolute values ​​of the vector elements. Matlab calls the function norm(x,2).

[0058] It should be understood that in optical microscopy, the diffraction pattern produced when an ideal point light source passes through the optical system is generally called an Airy disk. In light-field microscopy, the introduction of microlens arrays allows point light sources to produce more complex diffraction patterns. The Airy disk contains a wealth of three-dimensional information about the point light source, which forms the basis of 3D reconstruction algorithms. PSF information in light fields is obtained through translational changes. Specifically, the diffraction pattern behind the microlens array changes with the three-dimensional position of the point light source.

[0059] Specifically, the entire imaging process of a point light source in a light field microscopy system is as follows:

[0060] (1) First, simulate the process of light waves reaching the microscope objective. In this state, the first Born approximation can be used, assuming that there is no scattering in the volume. The radiation emitted from a certain point can be assumed to be a perfect spherical wave before reaching the microscope objective. When imaging an unobstructed or severely scattering object, the above approximation is valid. Using wave optics, the spherical wave in front of the microlens array can be analyzed as U(x, y, z):

[0061]

[0062]

[0063]

[0064] Where f0 and M are the focal length and magnification of the microscope objective, respectively; λ is the wavelength (i.e., the intensity of the point light source); α≈arcsin(NA / n), α is the maximum angle of incidence of the objective aperture, where n is the refractive index of the lens, NA is the numerical aperture of the objective, and J0 is the Bessel function.

[0065] (2) Secondly, when light waves pass through the microlens array, each lens can be equivalent to an amplitude mask represented by the lens aperture and a phase mask represented by the refraction of light through the lens itself:

[0066]

[0067] Wherein, D is the numerical aperture of the microlens; rect() is a rectangular function; s and t are the coordinates of the microlens array surface (ie, the microlens array surface coordinates).

[0068] The same amplitude and phase mask is extended to the rest of the incident wavefront, and the microlens array can also be described as the convolution of the two-dimensional comb function and Φ(x,y,z):

[0069]

[0070] The final light intensity distribution of the spherical wave emitted by the point light source S(x, y, z) in the imaging object is calculated using the Fourier transform operator:

[0071]

[0072] Where, ω v is the spatial frequency of the pixel at coordinate u on the sensor, ω u =u / lf mla ;ω v is the spatial frequency of the pixel with coordinate v on the sensor; and is the Fourier transform.

[0073] For optical systems, the PSF is the intensity distribution of the light field output when the input object is a point light source. Therefore, h(x, y, z) in the formula is the PSF of the light field microscopy system.

[0074] In the above embodiment, the light intensity distribution is calculated based on the PSF model for the point light source coordinates, point light source intensity and microlens array surface coordinates, which can obtain a large amount of three-dimensional information, making the three-dimensional reconstruction effect relatively good and suitable for three-dimensional reconstruction of light field microscopy.

[0075] Optionally, as an embodiment of the present invention, the process of calculating the estimated value of the light intensity distribution and the sub-aperture image to obtain the estimated value includes:

[0076] An estimated value is obtained by calculating the estimated value of the light intensity distribution and the sub-aperture image using the first formula and a preset number of iterations, wherein the first formula is:

[0077] f' i+1 (x,y,z)={g(x,y,z) / [f i '(x,y,z)*h(x,y,z)]*h(-x,-y,-z)}f i '(x,y,z),

[0078] Among them, f' i+1 (x, y, z) is the estimated value of the i+1th iteration, f i '(x,y,z) is the estimated value of the i-th iteration, g(x,y,z) is the subaperture image, f1(x,y,z)=g(x,y,z), h(x,y,z) is the light intensity distribution, and h(-x,-y,-z) is the conjugate representation of h(x,y,z).

[0079] It should be understood that the Lucy-Richardson algorithm is used to perform an iterative solution, that is, to perform deconvolution on the light intensity distribution of the object recorded in the image, to obtain an estimated value of the light intensity distribution of each point on the object.

[0080] Specifically, the 3D reconstruction model:

[0081] In light field microscopy, 3D reconstruction can be transformed into the following problem: using a known image sensor to record noisy light field microscopy information, the proposed algorithm is used to estimate the radiation intensity at each point on the observed target object. Thus, the following imaging model can be established, where the light field at the image sensor is:

[0082] g(x,y,z)=h(x,y,z)*f(x,y,z)+b

[0083] Where f(x, y, z) is the light field intensity distribution of the 3D reconstructed object; h(x, y, z) is the impulse response function; g(x, y, z) is the light field distribution recorded by the light field image; and b is the noise of the light field microscopy system.

[0084] 3D reconstruction iterative algorithm:

[0085] The model for microscopic 3D reconstruction is obtained above. However, due to diffraction of the optical microscope system and other reasons, the g(x, y, z) light field image recorded on the image plane is blurred and cannot be reconstructed in 3D. In this case, the Lucy-Richardson algorithm can be used to iteratively deconvolve the 3D object to obtain f(x, y, z) and achieve 3D reconstruction of the object.

[0086] The iterative formula of the Lucy-Richardson algorithm can be expressed as:

[0087]

[0088] in and are the estimated values ​​of the object point light intensity distribution obtained at the i-th and i+1-th iterations, respectively. That is f' i+1 (x,y,z), That is f i '(x,y,z).

[0089] Obviously, g(x,y,z) is known, but h(x,y,z) is unknown, so we build our own h(x,y,z) prediction model to predict h(x,y,z).

[0090] In the above embodiment, the estimated values ​​of the light intensity distribution and the sub-aperture image are calculated using the first formula and a preset number of iterations to obtain an estimated value, which can obtain a large amount of three-dimensional information, resulting in a relatively good three-dimensional reconstruction effect, and can be applied to three-dimensional reconstruction of light field microscopy.

[0091] Optionally, as an embodiment of the present invention, before constructing the microlens array surface coordinates through all microlens center coordinates, the method further includes the following steps:

[0092] Importing the distance between two microlenses corresponding to the center coordinates of each microlens, and respectively calculating the distance between the center coordinates of each microlens and the center coordinates of microlenses adjacent to the center coordinates of the microlens, to obtain the center coordinate distance corresponding to the center coordinates of each microlens;

[0093] It is determined whether each of the center coordinate distances is equal to the spacing between the two microlenses corresponding to the center coordinates of each microlens. If so, the microlens array surface coordinates are constructed using all the microlens center coordinates. If not, the midpoint of the spacing between the two microlenses corresponding to each unequal center coordinate distance is used as the microlens center coordinate corresponding to the center coordinate distance.

[0094] In the above embodiment, the center coordinates of each microlens and the distances between the center coordinates of adjacent microlenses are calculated to obtain the center coordinate distances. It is then determined whether each center coordinate distance is equal to the distance between two microlenses. If so, the microlens array surface coordinates are constructed using the center coordinates of all microlenses. If not, the midpoints of the distances between two microlenses corresponding to the unequal center coordinate distances are used as the center coordinates of the microlenses corresponding to the center coordinate distances, thereby improving the accuracy of subsequent three-dimensional reconstruction.

[0095] Alternatively, as another embodiment of the present invention, Figure 2 As shown, Figure 2 In the image, a micro lens array is installed at a focal length behind the objective lens, and the CCD camera is located at the rear focal plane of the micro lens array. The point light source is located in front of the front focal plane. Figure 2 The process of point light source in light field microscopy is demonstrated.

[0096] Optionally, as another embodiment of the present invention, the present invention first analyzes the process of incident light passing through the microscope objective lens, the microlens array, and reaching the image sensor based on wave optics; then, by modeling the microlens array, the point spread function (PSF) of the light field microscopy imaging system is obtained; finally, three-dimensional reconstruction is achieved through a deconvolution iterative algorithm.

[0097] Optionally, as another embodiment of the present invention, the present invention includes taking a white image and calculating the center point coordinates, obtaining a sub-aperture image, calculating the point spread function of each point of the microlens array, using a deconvolution algorithm for iterative solution, obtaining an image that satisfies the maximum likelihood estimation through iterative calculation, and outputting a super-resolution image. The image used for iterative reconstruction is the sub-view set extracted after calibration. The premise for the iteration is that the PSF, or impulse response function, of the microscopic imaging system is known. Therefore, according to the PSF calculation model provided by us, the PSF of the microscopic imaging system is calculated, and then the sub-view set is input into the Lucy-Richardson algorithm for iterative solution. Finally, light field microscopic images at different depths are output, completing the three-dimensional reconstruction of the object.

[0098] Optionally, as another embodiment of the present invention, two-dimensional super-resolution images of the object at different depths are output, thereby completing the three-dimensional reconstruction of the object.

[0099] Optionally, as another embodiment of the present invention, the present invention first calculates the PSF image of each point in the square, then substitutes it into the formula, solves iteratively, obtains the maximum likelihood estimate of the ideal image through iterative calculation, and then outputs the image, so as to realize three-dimensional reconstruction of the object.

[0100] Alternatively, as another embodiment of the present invention, the following beneficial effects are achieved: Based on wave optics theory, the propagation of light in a microscopic imaging system is analyzed, and a point spread function (PSF) calculation model is established. Using this PSF model, combined with 3D deconvolution theory, after multiple iterative calculations, light field microscopic images can be reconstructed in three dimensions. Experimental results demonstrate that the proposed method can obtain light field microscopic imaging results at different imaging depths, with relatively good 3D reconstruction results.

[0101] Figure 3 This is a module block diagram of a three-dimensional reconstruction device based on a microlens array provided by an embodiment of the present invention.

[0102] Alternatively, as another embodiment of the present invention, Figure 3 As shown, a three-dimensional reconstruction device based on a microlens array includes:

[0103] A white image processing module is used to import a plurality of light field raw data, perform white image processing on the plurality of light field raw data, and obtain a target white image;

[0104] An image screening module is used to screen the coordinates with the largest brightness value in the target white image, obtain multiple microlens center coordinates after screening, and construct the microlens array surface coordinates through all the microlens center coordinates;

[0105] The 3D reconstruction result acquisition module is used to obtain a sub-aperture image through a CCD camera, import point light source data, analyze the sub-aperture image, the point light source data and the microlens array surface coordinates to obtain an estimated value, and use the estimated value as the result of the 3D reconstruction.

[0106] Alternatively, another embodiment of the present invention provides a microlens array-based 3D reconstruction device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the microlens array-based 3D reconstruction method described above is implemented. The device may be a computer or other device.

[0107] Optionally, another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the three-dimensional reconstruction method based on the microlens array as described above is implemented.

[0108] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0109] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0111] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of the embodiments of the present invention.

[0112] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method based on a microlens array, characterized in that: The steps include: Importing a plurality of light field raw data, performing white image processing on the plurality of light field raw data to obtain a target white image; Screening the coordinates with the largest brightness value in the target white image, obtaining a plurality of microlens center coordinates after screening, and constructing the microlens array surface coordinates through all the microlens center coordinates; Obtaining a sub-aperture image through a CCD camera, importing point light source data, and analyzing the sub-aperture image, the point light source data, and the microlens array surface coordinates to obtain an estimated value, and using the estimated value as a result of three-dimensional reconstruction; The point light source data includes the point light source coordinates and the point light source intensity; The process of analyzing the sub-aperture image, the point light source data, and the microlens array surface coordinates to obtain an estimated value includes: Calculating the light intensity distribution of the point light source coordinates, the point light source intensity, and the microlens array surface coordinates based on the PSF model to obtain the light intensity distribution; Calculating estimated values ​​for the light intensity distribution and the sub-aperture image to obtain estimated values; The process of calculating the estimated value of the light intensity distribution and the sub-aperture image to obtain the estimated value includes: An estimated value is obtained by calculating the estimated value of the light intensity distribution and the sub-aperture image using the first formula and a preset number of iterations, wherein the first formula is: f' i+1 (x,y,z)={g(x,y,z) / [f i '(x,y,z)*h(x,y,z)]*h(-x,-y,-z)}f i '(x,y,z), Among them, f' i+1 (x, y, z) is the estimated value of the i+1th iteration, f i '(x,y,z) is the estimated value of the i-th iteration, g(x,y,z) is the subaperture image, f1'(x,y,z)=g(x,y,z), h(x,y,z) is the light intensity distribution, and h(-x,-y,-z) is the conjugate representation of h(x,y,z).

2. The three-dimensional reconstruction method based on a microlens array according to claim 1, characterized in that: The process of performing white image processing on the plurality of raw light field data to obtain a target white image includes: extracting a plurality of original white images corresponding to each of the light field original data respectively from each of the light field original data; Calibrate each of the extracted original white images to obtain a calibrated white image corresponding to each of the original white images; Performing denoising processing on each of the calibrated white images to obtain a denoised white image corresponding to each of the original white images; Perform mean processing on all denoised white images to obtain the target white image.

3. The three-dimensional reconstruction method based on a microlens array according to claim 1, characterized in that: The process of screening the coordinates with the largest brightness value in the target white image and obtaining the coordinates of the multiple microlenses centers after screening includes: The MATLAB tool is used to screen the coordinates with the maximum brightness value in the target white image, and after screening, multiple microlens center coordinates are obtained.

4. The three-dimensional reconstruction method based on a microlens array according to claim 1, characterized in that: The process of calculating the light intensity distribution based on the point light source coordinates, the point light source intensity, and the microlens array surface coordinates to obtain the light intensity distribution includes: The light intensity distribution is calculated based on the PSF model for the coordinates of the point light source, the intensity of the point light source, and the coordinates of the microlens array surface to obtain the light intensity distribution. The PSF model is: in, in, where α≈arcsin(NA / n), Among them, ω u1 is the spatial frequency of the pixel with coordinate u1 on the CCD camera, ω v1 is the spatial frequency of the pixel at coordinate v1 on the sensor, and where h(x,y,z) is the light intensity distribution, U(x,y,z) is the spherical wave of the microlens array, T(x,y,z) is the microlens array, Φ(x,y,z) is the phase mask, D is the numerical aperture of the microlens, s is the abscissa of the microlens array surface, t is the ordinate of the microlens array surface, comb is the two-dimensional comb function, f0 is the focal length of the microscope objective, M is the magnification, λ is the intensity of the point light source, α is the maximum angle of incidence of the objective aperture, J0 is the Bessel function, rect() is the rectangular function, || ||2 is the Euclidean norm, (x,y,z) are the coordinates of the point light source, n is the refractive index of the lens, and NA is the numerical aperture of the objective.

5. The three-dimensional reconstruction method based on a microlens array according to claim 1, characterized in that: Before constructing the microlens array surface coordinates through all the microlens center coordinates, the method further includes the following steps: Importing the distance between two microlenses corresponding to the center coordinates of each microlens, and respectively calculating the distance between the center coordinates of each microlens and the center coordinates of microlenses adjacent to the center coordinates of the microlens, to obtain the center coordinate distance corresponding to the center coordinates of each microlens; It is determined whether each of the center coordinate distances is equal to the spacing between the two microlenses corresponding to the center coordinates of each microlens. If so, the microlens array surface coordinates are constructed using all the microlens center coordinates. If not, the midpoint of the spacing between the two microlenses corresponding to each unequal center coordinate distance is used as the microlens center coordinate corresponding to the center coordinate distance.

6. A three-dimensional reconstruction device based on a microlens array, characterized in that: include: A white image processing module is used to import a plurality of light field raw data, perform white image processing on the plurality of light field raw data, and obtain a target white image; An image screening module is used to screen the coordinates with the largest brightness value in the target white image, obtain multiple microlens center coordinates after screening, and construct the microlens array surface coordinates through all the microlens center coordinates; a 3D reconstruction result acquisition module, configured to acquire a sub-aperture image through a CCD camera, import point light source data, analyze the sub-aperture image, the point light source data, and the microlens array surface coordinates to obtain an estimated value, and use the estimated value as the 3D reconstruction result; The point light source data includes the point light source coordinates and the point light source intensity; In the three-dimensional reconstruction result acquisition module, the process of analyzing the sub-aperture image, the point light source data, and the microlens array surface coordinates to obtain an estimated value includes: Calculating the light intensity distribution of the point light source coordinates, the point light source intensity, and the microlens array surface coordinates based on the PSF model to obtain the light intensity distribution; Calculating estimated values ​​for the light intensity distribution and the sub-aperture image to obtain estimated values; In the three-dimensional reconstruction result acquisition module, the process of calculating the estimated value of the light intensity distribution and the sub-aperture image to obtain the estimated value includes: An estimated value is obtained by calculating the estimated value of the light intensity distribution and the sub-aperture image using the first formula and a preset number of iterations, wherein the first formula is: f' i+1 (x,y,z)={g(x,y,z) / [f i '(x,y,z)*h(x,y,z)]*h(-x,-y,-z)}f i '(x,y,z), Among them, f' i+1 (x, y, z) is the estimated value of the i+1th iteration, f i '(x,y,z) is the estimated value of the i-th iteration, g(x,y,z) is the subaperture image, f1'(x,y,z)=g(x,y,z), h(x,y,z) is the light intensity distribution, and h(-x,-y,-z) is the conjugate representation of h(x,y,z).

7. A three-dimensional reconstruction system based on a microlens array, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the three-dimensional reconstruction method based on the microlens array according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the three-dimensional reconstruction method based on a microlens array according to any one of claims 1 to 5 is implemented.

Images