Microlens array calibration detection method and light field image processing method

Abstract

The invention belongs to the field of light field cameras, and discloses a micro-lens array calibration detection method and a light field image processing method. The calibration detection method comprises the following steps: acquiring an original image and graying the original image; processing by adopting an adaptive Otsu binarization algorithm to obtain a binary image; extracting a connected region after morphological denoising, and calculating the geometric center and area of each micro lens; obtaining an effective center point set through area screening; obtaining a grid main direction and an inclination angle based on principal component analysis; calculating an average interval and generating a theoretical array; forming a final punctuation set through rotation transformation alignment and adjacent distance screening; automatically judging whether calibration is qualified or not according to the detection coverage rate; and based on a qualified calibration result, high-quality digital refocusing is realized through light field data extraction, sub-aperture image generation, pixel offset calculation and weighted fusion. The problems that in the prior art, calibration precision is low, robustness is poor, self-evaluation ability is lacked, and a refocusing image is blurred are solved.

Description

Technical Field

[0001] This invention relates to the field of light field cameras, and more specifically, to a microlens array calibration and detection method and a light field image processing method. Background Technology

[0002] As an emerging computational imaging device, the light field camera differs from traditional cameras by adding a microlens array in front of the traditional imaging sensor, enabling it to simultaneously record the spatial position and direction information of light. This unique light field acquisition method gives the light field camera the ability to perform post-processing capabilities such as image capture and refocusing, perspective transformation, and 3D reconstruction, demonstrating enormous application potential in fields such as virtual reality, medical imaging, and industrial inspection.

[0003] Based on different light field acquisition principles, existing light field cameras are mainly divided into two categories: focusing light field cameras based on microlens arrays (such as the Lytro series) and multi-view acquisition systems based on camera arrays. Among them, light field cameras based on microlens arrays have become the mainstream choice in the consumer market due to their compact structure and relatively low cost. However, the imaging quality of this type of camera is highly dependent on the accurate calibration of the microlens array. Although existing calibration methods have been refined over many years, they still have many inherent defects, which seriously limit the actual imaging quality and engineering applicability of light field cameras.

[0004] Traditional feature-point-based calibration methods (such as Cho et al., 2013) propose using a checkerboard calibration board to estimate the center position of each microlens by detecting the projection of feature points under each microlens. While intuitive, this method requires extremely high precision in the fabrication of the calibration board and struggles to handle distortions in edge regions. The calibration toolbox developed by Dansereau et al. (2013) estimates array parameters by fitting elliptical features in microlens images, but this method suffers from a significant drop in accuracy at low resolutions or in the presence of optical aberrations.

[0005] Self-calibration methods based on light field analysis (such as Zhang et al., 2016) propose to use the angular consistency constraint of the light field for self-calibration. This method does not require a specific calibration target, but it has high computational complexity and is sensitive to initial values.

[0006] Methods based on the assumption of regular grids cannot cope with array distortion, rotation and local transformation in actual processing, and have extremely poor robustness; feature extraction methods based on image processing are limited by image quality and their performance is severely degraded under blur, noise or light crosstalk interference; while methods based on optimization algorithms can achieve high accuracy under certain conditions, they are difficult to use due to huge computational overhead or rely on a large amount of labeled data, resulting in insufficient generalization ability.

[0007] Of particular concern are that existing methods generally have lower accuracy than those used in the central region, are not well adapted to low signal-to-noise ratio images, struggle to handle optical crosstalk between microlenses, and lack a unified standard for evaluating accuracy.

[0008] Meanwhile, existing methods also face significant challenges in the core of light field image post-processing—the refocusing algorithm. Geometric model-based refocusing methods, such as the "shift-overlay" algorithm, achieve focal plane changes by translating and aligning sub-aperture images and then weighted and merging them. These methods are simple to implement and computationally inexpensive, but they have limitations in refocusing accuracy and image sharpness, especially in areas with edges and complex textures where aliasing and blurring are prone to occur.

[0009] The refocusing method based on frequency domain transformation: Fourier transform is used to process the light field image, and filtering and reconstruction are performed in the frequency domain. It has high focus adjustment accuracy, but the algorithm has high complexity and is sensitive to noise, which is not conducive to real-time processing.

[0010] Depth estimation-based refocusing methods reconstruct the scene's depth map by calculating the parallax between different viewpoints in the light field image, and then synthesize the focused image using a remapping algorithm. These methods are suitable for scenes with clear geometric structures, but their depth estimation accuracy is poor in areas with smooth textures and under complex lighting conditions.

[0011] Deep learning-based refocusing methods: In recent years, deep learning models such as Convolutional Neural Networks (CNNs) and Transformers have been widely applied to the field of light field image processing. Related research utilizes neural networks to perform end-to-end refocusing or depth map generation on input light field maps, effectively improving image sharpness and focusing accuracy, and exhibiting strong generalization capabilities. However, these methods are typically highly dependent on training data, resulting in high computational resource consumption and poor algorithm interpretability.

[0012] In summary, existing microlens array calibration and light field image processing methods have significant shortcomings in terms of accuracy, robustness, automation, and computational efficiency. There is an urgent need for an innovative technical solution that can provide an integrated solution for high-precision, high-robustness, and self-evaluation calibration of microlens arrays, and on this basis, achieve high-quality light field refocusing. Summary of the Invention

[0013] The purpose of this invention is to overcome the shortcomings of the prior art and provide a microlens array calibration and detection method and a light field image processing method.

[0014] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0015] In a first aspect, the present invention provides a microlens array calibration and detection method, the method comprising the following steps:

[0016] Step S1: Obtain the original image containing microlens array imaging information and perform grayscale processing to obtain a grayscale image;

[0017] Step S2: The grayscale image is processed using an adaptive binarization algorithm based on the Otsu method to obtain a binary image;

[0018] Step S3: Perform morphological denoising on the binary image to remove noisy regions;

[0019] Step S4: Extract the connected components in the denoised image and calculate the geometric center point and area of ​​each connected region to obtain the geometric center point and area value of each microlens in the microlens array.

[0020] Step S5: Preset the area value filtering range, and extract the geometric center points whose area values ​​are within the preset area value filtering range as the effective center point set;

[0021] Step S6: Perform principal component analysis on the effective center point set to obtain the principal direction and tilt angle of the grid formed by the effective center point set;

[0022] Step S7: Calculate the average distance between all geometric center points using the nearest neighbor algorithm;

[0023] Step S8: Generate a theoretical microlens array based on the average spacing between the principal direction and the geometric center point;

[0024] Step S9: Rotate the actual microlens array according to the tilt angle to align it with the theoretical microlens array, and obtain the final punctuation set; wherein, the final punctuation set includes:

[0025] a) The actual geometric center point that coincides with the center point in the theoretical microlens array;

[0026] b) Remove geometric center points whose nearest neighbor distance to surrounding geometric center points is greater than a set threshold;

[0027] Step S10: Calculate the ratio of the number of geometric center points in the final set of markers to the total number of center points in the theoretical microlens array to obtain the detection coverage.

[0028] If the detection coverage is greater than or equal to the set coverage, the calibration is deemed qualified.

[0029] If the detection coverage is less than the set coverage, the calibration is deemed unqualified.

[0030] Preferably, step S2 is as follows:

[0031] Step S2.1: Count the number of pixels at each gray level i, and denote it as histogram n. i , i∈[0, L-1];

[0032] Step S2.2: Calculate the probability distribution p of each gray level. i :

[0033] p i = n i N , i∈[0, L-1]

[0034] Where N = n0 + n1 + ... + n i +…+n （L-1） This represents the total number of pixels.

[0035] Step S2.3, based on probability distribution p i Iterate through all thresholds t from 0 to L-1;

[0036] Step S2.4: Iterate through all thresholds t and select the optimal threshold T;

[0037] Step S2.5: Binarize the entire grayscale image using the optimal threshold T;

[0038] Step S2.6: Set the sensitivity parameter Sensitivity, 0.2 < Sensitivity < 0.6, and repeat steps S2.1-S2.5 to obtain the binary image.

[0039] Preferably, in step S5, filtering is performed according to a preset area value filtering range, wherein the upper limit Smax and the lower limit Smin of the filtering range are determined according to the following formula:

[0040]

[0041]

[0042] Where R represents the diameter of the microlens.

[0043] Preferably, step S6 specifically includes:

[0044] Step S6.1: Centralize the data from the effective centroids:

[0045] , i, j∈[0, L-1]

[0046] in, This represents the result of centering the x-coordinates of the geometric center points in the effective center point set. This represents the result of centering the ordinates of the geometric center points in the effective center point set; , These represent the x and y coordinates of several center points, respectively.

[0047] Step S6.2, shift the centered data to a mean of zero:

[0048]

[0049] in, Let be the relative center coordinates, representing the zero-mean matrix after the overall translation of the effective center point set;

[0050] Step S6.3, calculate the covariance matrix of the translated zero-mean matrix:

[0051]

[0052] in, Represent the covariance matrix; Indicates the variance in the x-direction; = Represents covariance; Indicates the variance in the y-direction;

[0053] Step S6.4, for the covariance matrix Perform eigenvalue decomposition:

[0054]

[0055] in, Represents eigenvalues. represents the eigenvector; the eigenvector corresponding to the largest eigenvalue represents the principal direction, and the eigenvector corresponding to the smallest eigenvalue represents the secondary direction.

[0056] Step S6.5, calculate the mesh tilt angle:

[0057]

[0058] in, , These are the elements of the eigenvector corresponding to the largest eigenvalue.

[0059] Preferably, step S7 specifically includes: step S7.1, for each point ( Find its nearest neighbor and calculate the Euclidean distance. :

[0060] , i, j∈[0, L-1]

[0061] Step S7.2, based on Euclidean distance Calculate the average spacing between geometric center points :

[0062] .

[0063] Preferably, the threshold set in step S9 is 1.3-1.7 times the average spacing between geometric center points.

[0064] Preferably, in step S10, the value range of the set threshold is 95%-98%.

[0065] Secondly, the present invention provides a light field image processing method based on a refocusing algorithm, the method comprising the following steps:

[0066] Step (1): The microlens array of the light field camera is calibrated using the microlens array calibration and detection method described above. If the calibration is qualified, the following steps are performed.

[0067] Step (2): Obtain the original image containing microlens array imaging information and perform grayscale processing to obtain a grayscale image, which is expressed as follows:

[0068] I x ,y , x ∈ 1, W , y ∈ [1, H ];

[0069] Where W and H are the width and height of the original CCD image;

[0070] Step (3): The grayscale image is segmented according to the geometric parameters of the microlens array, and the sub-images corresponding to each microlens are extracted to construct the four-dimensional light field data corresponding to each microlens:

[0071]

[0072] m∈[1,M],n∈[1,N],u∈[1,U],v∈[1,V]

[0073] Where m is the row of the microlens array, n is the column of the microlens array; u is the sub-image row, v is the sub-image column;

[0074] Step (4): Arrange the four-dimensional light field data according to the structure of the microlens array to generate complete light field cube data containing spatial dimensions (m,n) and angular dimensions (u,v):

[0075]

[0076] in, Indicates rearrangement;

[0077] Step (5): Under a fixed angular direction (u0, v0), extract all pixels at that angle to obtain the sub-aperture image.

[0078]

[0079] Step (6): Determine the position of the virtual refocusing plane according to the lens imaging formula, and calculate L1:

[0080]

[0081] Where f is the focal length of the main lens, U1 is the original reference plane distance, U2 is the target focusing plane distance, L is the original phase distance, and L1 is the target image distance;

[0082] And calculate the geometric scaling factor: ;

[0083] Step (7), calculate the pixel offset of the sub-aperture image:

[0084]

[0085] Where: Δ represents the pixel offset spacing, Δ=D·p, D is the normalized scaling factor, 1<D<100, and p is the physical size of a single pixel of the camera sensor;

[0086] Normalized to pixel offset: ;

[0087] Step (8): Offset compensation is performed on the fixed-angle view, and weighted fusion is performed to obtain the refocused image:

[0088]

[0089] in, , W is the weighting function.

[0090] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the microlens array calibration and detection method or the light field image processing method.

[0091] The present invention has the following beneficial effects: The present invention effectively solves the problems of low calibration accuracy, poor robustness, lack of self-evaluation ability and blurred refocusing images in the prior art, and significantly improves the automation level of microlens array calibration and the imaging quality of light field images.

[0092] This invention provides a microlens array calibration and detection method that achieves:

[0093] 1. Improved detection accuracy and robustness: Through adaptive threshold segmentation and morphological processing, it maintains high detection stability under different background conditions and is suitable for various microlens image types.

[0094] 2. Strong structural integrity recovery capability: Through average spacing estimation and grid point inference mechanisms, it effectively fills in missing lens areas, improves array recognition completeness, and adapts to imperfect imaging scenarios. By utilizing area and connected component attributes, it improves screening accuracy and reduces false positive rates.

[0095] 3. Strong adaptive correction capability for mesh orientation: By using principal component analysis to estimate the mesh orientation, it can automatically adapt to microlens arrays arranged at different rotation angles, eliminating the need for manual calibration.

[0096] 4. It has a self-evaluation function: It introduces the detection ratio index to realize the automatic quantitative evaluation of the results, which facilitates the monitoring of algorithm effect and quality control.

[0097] 5. Automated operation and suitable for batch image processing: This method requires no manual intervention and is suitable for embedding in automated processes such as large-scale image acquisition, industrial vision inspection, and optical system debugging to achieve high-quality light field image processing.

[0098] This invention provides a light field image processing method based on the calibration method, which achieves:

[0099] 1. Clearer Focused Images: By combining microlens array calibration and detection methods and introducing microlens offset, and using subpixel translation and weighted interpolation fusion, the reconstructed images at multiple object distances are more layered and can clearly distinguish structural features at different depths.

[0100] 2. More efficient use of information: By integrating spatial and angular image information, the multi-view images acquired by microlenses are fully utilized, improving the detail reproduction of the refocused images.

[0101] 3. Strong adaptability and high computational stability: The algorithm is highly robust to image noise and geometric deviations, and can be extended to multiple scenarios such as combustion field reconstruction, 3D visual inspection, and industrial measurement. Attached Figure Description

[0102] Figure 1 This is a comparison image of the microlens array in Example 1 before and after rotation at a tilt angle;

[0103] Figure 2 This is a magnified view of the local microlens array to be calibrated;

[0104] Figure 3 A magnified view of the microlens array calibration;

[0105] Figure 4 This is a diagram showing the arrangement of sub-aperture images;

[0106] Figure 5 This is a schematic diagram of the optical system of a light field camera;

[0107] Figure 6 For comparison of the visualized images obtained after microlens calibration in Comparative Example 1 and Example 1;

[0108] Figure 7 These are comparison images of the images refocused at different positions obtained in Example 2;

[0109] Figure 8 Comparison images obtained from Example 2, showing refocusing at different positions;

[0110] Figure 9 This is a comparison chart of the calibration results and interpolation results of Example 1 (Method 1) and Comparative Example 1 (Method 2);

[0111] Figure 10 This is the refocusing result when D=1;

[0112] Figure 11 This is the refocusing result when D=100. Detailed Implementation

[0113] To make the present invention easier to understand, the present invention will be further described below with reference to specific embodiments and accompanying drawings. These embodiments and drawings are not intended to limit the present invention in any way. They are only used to illustrate the present invention and are not intended to limit the scope of the present invention. Any modifications or changes to the present invention that are easily implemented by those skilled in the art without departing from the technical solutions of the present invention will fall within the scope of the claims of the present invention.

[0114] An optical system for a light field camera, comprising:

[0115] Main lens: It is the first component that allows light to enter the system, and it is responsible for collecting and initially focusing the light.

[0116] Microlenses; installed between the main lens and the relay lens, used to further process light and change its propagation characteristics;

[0117] Relay mirror: Located after the microlens, it relays and refocuses the light rays processed by the microlens;

[0118] Image sensor: Installed at the end of the system, it is used to receive light after it has been processed by the optical components in front, and convert the light signal into an electrical signal, thereby forming image data; the components are connected coaxially in sequence through a mechanical structure to ensure that the light passes through each component in sequence along a straight propagation path.

[0119] A microlens array calibration and detection method includes the following steps:

[0120] Step S1: Using the light field camera optical system described above, acquire the original image containing microlens array imaging information and perform grayscale processing to obtain a grayscale image; the acquisition conditions ensure that the image is not overexposed.

[0121] Specifically: Weighted average of the three color channels: gray(x,y)=0.2989 R + 0.5870 G + 0.1140 B B;

[0122] Step S2: Apply an adaptive local thresholding algorithm (with sensitivity parameter adjustment) based on the Otsu method to the grayscale image to obtain a binary image;

[0123] Step S2.1, calculate the histogram: for the entire grayscale image, the pixel range [0, L-1] is calculated, and the number of pixels at each grayscale level i is counted, denoted as histogram n. i The total number of pixels is N = n0 + n1 + ... + n i +…+n （L-1） In this embodiment, L=4096;

[0124] Step S2.2, calculate the probability distribution: the probability of each gray level appearing is p. i :

[0125] p i = n i N , i ∈ [ 0,L -1]

[0126] Step S2.3: Iterate through all possible thresholds t (from 0 to L-1).

[0127] Step S2.3.1, calculate the cumulative probability w0(t) of the background and the cumulative probability w1(t) of the foreground:

[0128]

[0129] Step S2.3.2: Calculate the average gray level μ0(t) of the background and the average gray level μ1(t) of the foreground:

[0130]

[0131] Step S2.3.3, calculate the inter-class variance:

[0132]

[0133] Step S2.4: Select the optimal threshold T

[0134] By iterating through all thresholds t, we find the threshold t that maximizes the inter-class variance, i.e., the optimal threshold T is:

[0135]

[0136] Step S2.5, Binarization Process

[0137] Binarize the entire grayscale image using the optimal threshold T:

[0138] ;

[0139] in, Represents the pixel grayscale value;

[0140] Step S2.6: Set the sensitivity parameter Sensitivity, 0.2 < Sensitivity < 0.6, as the control parameter in the local adaptive thresholding algorithm. Repeat steps S2.1-S2.5 to ensure that the center point of the microlens can still be stably detected under uneven illumination or local noise conditions, thus obtaining a binary image. In the implementation of this invention, the sensitivity parameter Sensitivity in the local adaptive thresholding segmentation algorithm was systematically adjusted and verified. This parameter is used to adjust the tightness of the local threshold and has a significant impact on the segmentation accuracy of the microlens region. When the parameter value is too low, some microlens regions cannot be correctly segmented; while when the parameter value is too high, noise is easily introduced.

[0141] Step S3: Perform morphological denoising on the binary image to remove noisy regions;

[0142] Specifically, the binary image is first eroded, then dilated, and isolated noise points are removed by "opening" operation. All possible microlens regions are extracted by combining connected component analysis.

[0143] Step S4: Extract the connected regions of the microlens array from the denoised image and calculate the geometric center point and area value of each microlens in the microlens array.

[0144] Step S5: Set the area value filtering range, and extract the geometric center points of the area values ​​within the set area value filtering range as the effective center point set, i.e. ;

[0145] The area value filtering range is determined based on the following:

[0146]

[0147]

[0148] in, Indicates the upper limit of the area value filtering range; Indicates the lower limit of the area value filtering range; R represents the microlens diameter;

[0149] Step S6: Perform principal component analysis based on the effective center point set to extract the grid tilt angle;

[0150] Step S6.1: Centralize the data in the effective centroid set:

[0151]

[0152] in, This represents the result of centering the x-coordinates of the geometric center points in the effective center point set. This represents the result of centering the ordinates of the geometric center points in the effective center point set;

[0153] Step S6.2, shift the centered data to a mean of zero:

[0154]

[0155] in, Let be the relative center coordinates, representing the zero-mean matrix after the overall translation of the effective center point set;

[0156] Step S6.3, calculate the covariance matrix:

[0157]

[0158] in, Represent the covariance matrix; Indicates the variance in the x-direction; = Represents covariance; Indicates the variance in the y-direction;

[0159] Step S6.4, for the covariance matrix Perform eigenvalue decomposition:

[0160]

[0161] in, Represents eigenvalues. This represents the eigenvectors; the eigenvector corresponding to the largest eigenvalue represents the principal direction (the direction of point cloud extension), and the eigenvector corresponding to the smallest eigenvalue represents the secondary direction (the vertical direction).

[0162] Step S6.5, calculate the mesh tilt angle:

[0163]

[0164] in, , These are the elements of the eigenvector corresponding to the largest eigenvalue;

[0165] Step S7: Calculate the average distance between all geometric center points using the nearest neighbor algorithm;

[0166] Step S7.1, for each point ( Find its nearest neighbor and calculate the Euclidean distance. :

[0167]

[0168] Step S7.2, based on Euclidean distance Calculate the average spacing between geometric center points :

[0169]

[0170] Step S8: Based on the principal direction of the geometric center points in the effective center point set and the average spacing between the geometric center points... Generate a theoretical microlens array;

[0171] Step S9: Align the obtained microlens array with the theoretical microlens array according to the grid tilt angle rotation transformation. First, retain the geometric center points that coincide with the theoretical microlens array. Then, remove the geometric center points whose nearest neighbor distance to the surrounding geometric center points is greater than a set threshold to form the final mark set.

[0172] The threshold setting is related to the noise of the microlens. The lower the noise, the smaller the multiple of the average distance; the higher the noise, the larger the multiple of the average distance. Specifically, it is 1.3-1.7 times the average distance, and in this embodiment, it is 1.5 times the average distance.

[0173] Step S10: Calculate the ratio of the number of calibration points in the final calibration point set to the total number of theoretical microlens array center points, i.e., the detection coverage rate. If the detection coverage rate is greater than or equal to the set coverage rate, the calibration is completed. If the detection coverage rate is less than the set coverage rate, the calibration is unqualified.

[0174] The coverage setting is related to the accuracy requirements of the detection task and the application scenario:

[0175] 1. Detection accuracy requirements

[0176] For scientific research or high-precision optical applications (such as light field microscopy and 3D imaging), a coverage of ≥ 98% is usually required to ensure that the calibration is almost complete.

[0177] For routine manufacturing inspections, 95% is already considered statistically acceptable.

[0178] 2. Noise level and image quality

[0179] High noise levels can easily lead to missed detections and lower coverage, so the threshold may be appropriately relaxed. Clear images will have fewer missed detections, so a higher coverage can be required.

[0180] 3. Total size of the microlens array

[0181] If the total number is large (e.g., tens of thousands of points), even a small number of defects will not affect the overall acceptable 95%; if the array size is small, the defect ratio will have a greater impact and may need to be ≥ 98%.

[0182] Step S11: Overlay the detection points and inferred points to form a clear and intuitive image;

[0183] Step S12: Generate an image that only displays the detection points for manual review and debugging.

[0184] A light field image processing method based on a refocusing algorithm includes the following steps:

[0185] Step (1): Use the above-mentioned microlens array calibration and testing method to evaluate whether the microlens array calibration is qualified. If it is qualified, proceed to the following steps. If it is not qualified, recalibrate and evaluate.

[0186] Step (2): Using the calibrated light field camera optical system, acquire the original image containing microlens array imaging information, and perform grayscale processing to obtain a grayscale image:

[0187] I x ,y , x ∈ 1, W , y ∈ [1, H ];

[0188] in,( , ) represents the pixel coordinates of the original image, and W and H represent the width and height of the original image (4096×4096 in this embodiment).

[0189] Step (3): Based on the geometric parameters of the microlens array (original image size, microlens array distribution), the grayscale image is divided into blocks using the physical spacing d of the microlenses. Sub-images corresponding to each microlens are extracted, and four-dimensional light field data corresponding to each microlens is constructed. The spatial and angular information of the light is recorded, that is, local four-dimensional light field segments (2D to 4D index) are extracted from the two-dimensional grayscale image.

[0190]

[0191] m∈[1,M],n∈[1,N],u∈[1,U],v∈[1,V]

[0192] Where u is the sub-image row, v is the sub-image column; m is the row in the microlens array, n is the column in the microlens array; U is the width of the sub-image, V is the height of the sub-image, M is the width of the microlens array, and N is the height of the microlens array.

[0193] Step (4): Arrange the local four-dimensional light field fragments of each microlens extracted in step (3) according to the structure of the microlens array, and rearrange the data so that all spatial points (m,n) at the same angle (u,v) are organized together to generate complete light field cube data containing spatial dimension (m,n) and angular dimension (u,v). (Create a new 4D data structure):

[0194]

[0195] This indicates a rearrangement operation, which copies the values ​​of the original data L(m,n,u,v) into the memory according to the priority order of (u,v,m,n). This changes the underlying data storage order, so that when the two dimensions (u,v) are fixed, the data is contiguous in memory, thus enabling efficient extraction of sub-aperture images.

[0196] Step (5): Under a fixed angular direction (u0, v0), extract all pixels at that angle to obtain the sub-aperture image.

[0197]

[0198] Received It is a sub-aperture image, which changes ( , This allows the generation of sub-aperture images at different angles.

[0199] Step (6): Determine the position of the virtual refocusing plane according to the lens imaging formula, and calculate L1:

[0200]

[0201] Where f is the focal length of the main lens, U1 is the original reference plane distance, U2 is the target focusing plane distance, L is the original phase distance, and L1 is the target image distance;

[0202] And calculate the geometric scaling factor: .

[0203] Step (7), calculate the pixel offset of the sub-aperture image:

[0204] Where: Δ represents the pixel offset spacing, Δ = D·p, D is the normalized scaling factor, 1 < D < 100. Within this range, while ensuring significant parallax effect, image misalignment and blurring can be avoided, thereby obtaining a clear and stable refocused image. p is the physical size of a single pixel of the camera sensor.

[0205] Normalized to pixel offset: .

[0206] Step (8): Offset compensation is performed on each angle view, and weighted fusion is performed to obtain the refocused image. All sub-aperture images are then merged using the shift stacking method.

[0207]

[0208] in, , ; This is a weighting function used to count the number of pixel accumulations to obtain the refocused image under the target focal plane. .

[0209] Example 1 A microlens array calibration and detection method includes the following steps: Step S1: Obtain the original image containing microlens array imaging information and perform grayscale processing to obtain a grayscale image; Step S2: Apply an adaptive binarization algorithm (with sensitivity parameter adjustment) based on the Otsu method to the grayscale image and apply an adaptive local threshold to obtain a binary image. The sensitivity parameter Sensitivity for adaptive binarization is set to 0.4. When Sensitivity = 0.2, the number of detected microlens center points is 23028, with a coverage ratio of 82.24%; when Sensitivity = 0.4, the number of detected microlens center points is 27084, with a coverage ratio of 96.73%; and when Sensitivity = 0.6, the number of detected microlens center points is 27063, with a coverage ratio of 96.65%. The number of detections increased significantly from 0.2 to 0.4 (+17.6%), while the number of detections remained basically flat from 0.4 to 0.6 (-0.08%). 0.4 has reached the performance saturation point, and further increasing the sensitivity is meaningless. Although the number of detections at 0.6 is similar to that at 0.4, it may introduce false positives: oversensitivity may misjudge noise as microlens boundary effects: incomplete or edge microlenses may be detected.

[0210] Step S3: Perform morphological denoising on the binary image to remove noisy regions; Step S4: Extract the connected regions of the microlens array from the denoised image and calculate the geometric center point and area value of each microlens in the microlens array. Step S5: Set the area value filtering range to 100-500 pixels. 2 Extract the geometric center points whose area values ​​fall within the set area value filtering range as the effective center point set; Step S6: Perform principal component analysis based on the effective center point set to extract the grid tilt angle; Step S7: Calculate the average distance between all geometric center points using the nearest neighbor algorithm; Step S8: Based on the principal direction of the geometric center points in the effective center point set and the average spacing between the geometric center points... Generate theoretical microlens arrays; Step S9: Align the obtained microlens array with the theoretical microlens array according to the grid tilt angle rotation transformation, such as... Figure 1 As shown, firstly, the geometric center points that coincide with the theoretical microlens array are retained. Then, by removing geometric center points whose nearest neighbor distance to the surrounding geometric center points is greater than 1.5 times the average distance, the final set of punctuation points is formed, with a total number of 27084. Step 10, coverage rate is 95%, theoretical total number of microlens array center points is 28000, output detection coverage rate is 96.73%.

[0211] Comparative Example 1 Compared with Example 1, step 2 uses a fixed threshold; step 5 relies solely on area value filtering, directly outputting the number of geometric center points within the area value filtering range as 15605; step 9 relies solely on the original detection points and scatter interpolation.

[0212] like Figure 6 As shown, the left side is the image visualized after calibration in Comparative Example 1, and the right side is the image visualized after calibration in the Example. Calibration is performed using the method of this invention. The overall recognition efficiency is evaluated by the detection ratio, allowing for a more intuitive and comprehensive observation of the calibration results. Furthermore, the screening is more accurate, with a low false positive rate. Before using Otus adaptive segmentation and morphological processing, the detection accuracy was only about 56%.

[0213] likeFigure 9 As shown, the top left corner shows the point distribution of Example 1 (red circles = detection points, green "+" = inference points), the top right corner shows the regular and uniform interpolation result of Example 1, the bottom left corner shows the point distribution of Comparative Example 1 (blue circles = original detection points), and the bottom right corner shows the irregular interpolation result of Comparative Example 1 with distortion. Example 1 uses a mesh fitting and interpolation method to fill in missing points. The mesh structure is uniform, which can fill in the missing detection points. It is more robust to noise and missing points, and the distribution after interpolation is more regular, which is suitable for subsequent calibration and reconstruction. The scattered interpolation method in Comparative Example 1 is greatly affected by the distribution of detection points, resulting in significant interpolation distortion. It is also sensitive to missing points and noise, and exhibits poor regularity.

[0214] Example 2

[0215] A light field image processing method based on a refocusing algorithm includes the following steps: Step (1): Use the above-mentioned microlens array calibration and testing method to evaluate whether the microlens array calibration is qualified. If it is qualified, proceed to the following steps. If it is not qualified, recalibrate and evaluate. Step (2): Using the calibrated optical system of the light field camera, acquire the original image containing microlens array imaging information (4096×4096 resolution, where the microlens array is arranged in 160×160, and the sub-image size corresponding to each microlens is 25×25 pixels), and perform grayscale processing to obtain a grayscale image. Step (3): Based on the geometric parameters of the microlens array, the grayscale image is divided into blocks using the physical spacing d of the microlenses. Sub-images corresponding to each microlens are extracted, and four-dimensional light field data corresponding to each microlens is constructed. The spatial and angular information of the light is recorded, that is, local four-dimensional light field segments are extracted from the two-dimensional grayscale image; where M and N are 160, and U and V are 25. Step (4): Arrange the local four-dimensional light field fragments of each microlens extracted in step (3) according to the structure of the microlens array to generate complete light field cube data containing spatial dimensions (m,n) and angular dimensions (u,v). ; Step (5): Under a fixed angular direction (u0, v0), extract all pixels at that angle to obtain the sub-aperture image; Step (6): Determine the position of the virtual refocusing plane according to the lens imaging formula, calculate L1, and calculate the geometric scaling factor: ; Step (7): Calculate the pixel offset of the sub-aperture image; normalize it to pixel offset, where the physical size p of a single pixel of the camera sensor is 0.0055.

[0216] Introducing three image quality rating functions: MSE (Mean Squared Error) ; Where M×N is the image size. The reference image has a pixel value at position (i,j). Let be the pixel value of the image to be evaluated at position (i,j); its physical meaning is to measure the degree of difference between the two images at the pixel level.

[0217] PSNR (Peak Signal-to-Noise Ratio): ; Where MSI is the maximum possible pixel value of the image, and MSE is the mean square error; physically, it measures the ratio of image signal to noise (error).

[0218] SSIM (Structural Similarity Index): ; Where μ x ,μ y σ represents the mean (brightness) of the image at x and y coordinates. x 2 ,σ y 2 σ represents the variance (contrast) of the image x and y. xy Let C1 and C2 be the covariance (structure) of the images x and y, and C1 and C2 be stability constants; their physical meaning is more in line with the perceptual quality evaluation of the human visual system.

[0219] All three metrics are image evaluation functions, but MSE / PSNR are traditional objective metrics, simple to calculate but not in line with human visual perception. SSIM, on the other hand, is a modern subjective perception metric, more in line with actual quality perception. For refocusing experiments, it is recommended to use a combination of primarily referencing SSIM (which best reflects visual quality), supplemented by PSNR (a traditional standard), and monitoring MSE (to understand pixel-level errors) to comprehensively evaluate the quality of refocused images.

[0220] Therefore, the acceptable quality range is set as MSE ≤ 30, SSIM ≥ 0.95, and PSNR ≥ 33 dB.

[0221] like Figure 10 As shown, when D=1, the optimal focal lengths are 190mm & 210mm (MSE=0, SSIM=1, PSNR=Inf). The characteristic is near-perfect overlap with zero error. The problem is that it is too sensitive and lacks practical application value.

[0222] When D=100, the optimal focal length is 210mm, the MSE range is 203.32-1193.19, the SSIM range is 0.6415-0.9039, and the PSNR range is 17.36-25.05 dB. A key characteristic is the large error and significant degradation in image quality.

[0223] When D=10, the optimal focal length is 210mm, with an MSE range of 10.19-112.19, an SSIM range of 0.9231-0.9731, and a PSNR range of 27.63-38.05 dB. Its characteristics include good resolution and a reasonable error range.

[0224] At D=10: 1. It exhibits moderate sensitivity, clearly distinguishing the effects of different refocusing distances. The reasonable error range is neither too sensitive nor too lenient, and the stable optimal focus clearly points to 210mm, consistent with theoretical expectations. 2. Image quality is guaranteed: MSE range of 10.19-112.19, SSIM range of 0.9231-0.9731, and PSNR range of 27.63-38.05 dB. 3. It has practical application value, maintaining stable evaluation results under different conditions, effectively distinguishing images close to the optimal focus, and conforming to the performance characteristics of practical light field imaging systems. The preferred value for this invention is D=10.

[0225] Step (8): Offset compensation is performed on each angle view, and weighted fusion is performed to obtain the refocused image. All sub-aperture images are then merged using the shift stacking method. ; in, , W=1 (uniform weighting, suitable for refocusing sharpness optimization).

[0226] Comparative Example 2 Compared with Example 2, step (1) is missing, and in step (7), the pixel offset of the sub-aperture image is obtained by floating-point + bilinear interpolation.

[0227] like Figure 7 and Figure 8 As shown, the refocusing method of the present invention produces more accurate and clearer images, and effectively improves operating efficiency.

[0228] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. However, the above description is merely a specific embodiment of the present invention, and the technical features of the present invention are not limited thereto. Any other embodiments derived by those skilled in the art without departing from the technical solution of the present invention should be covered within the patent scope of the present invention.

Claims

1. A method for calibrating and detecting a microlens array, characterized in that, The method includes the following steps: Step S1: Obtain the original image containing microlens array imaging information and perform grayscale processing to obtain a grayscale image; Step S2: The grayscale image is processed using an adaptive binarization algorithm based on the Otsu method to obtain a binary image; Step S3: Perform morphological denoising on the binary image to remove noisy regions; Step S4: Extract the connected components in the denoised image and calculate the geometric center point and area of ​​each connected region to obtain the geometric center point and area value of each microlens in the microlens array. Step S5: Preset the area value filtering range, and extract the geometric center points whose area values ​​are within the preset area value filtering range as the effective center point set; Step S6: Perform principal component analysis on the effective center point set to obtain the principal direction and tilt angle of the grid formed by the effective center point set; Step S7: Calculate the average distance between all geometric center points using the nearest neighbor algorithm; Step S8: Generate a theoretical microlens array based on the average spacing between the principal direction and the geometric center point; Step S9: Rotate the actual microlens array according to the tilt angle to align it with the theoretical microlens array, and obtain the final punctuation set; wherein, the final punctuation set includes: a) The actual geometric center point that coincides with the center point in the theoretical microlens array; b) Remove geometric center points whose nearest neighbor distance to surrounding geometric center points is greater than a set threshold; Step S10: Calculate the ratio of the number of geometric center points in the final set of markers to the total number of center points in the theoretical microlens array to obtain the detection coverage. If the detection coverage is greater than or equal to the set coverage, the calibration is deemed qualified. If the detection coverage is less than the set coverage, the calibration is deemed unqualified.

2. The microlens array calibration and detection method according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S2.1: Count the number of pixels at each gray level i, and denote it as histogram n. i , i∈[0, L-1]; Step S2.2: Calculate the probability distribution p of each gray level. i : ； Where N = n0 + n1 + ... + n i +…+n （L-1） This represents the total number of pixels. Step S2.3, based on probability distribution p i Iterate through all thresholds t from 0 to L-1; Step S2.4: Iterate through all thresholds t and select the optimal threshold T; Step S2.5: Binarize the entire grayscale image using the optimal threshold T; Step S2.6: Set the sensitivity parameter Sensitivity, 0.2 < Sensitivity < 0.6, and repeat steps S2.1-S2.5 to obtain the binary image.

3. The microlens array calibration and detection method according to claim 1, characterized in that, In step S5, filtering is performed according to a preset area value filtering range, wherein the upper limit Smax and lower limit Smin of the filtering range are determined according to the following formula: ； ； Where R represents the diameter of the microlens.

4. The microlens array calibration and detection method according to claim 1, characterized in that, Step S6 is as follows: Step S6.1: Centralize the data from the effective centroids: ,i,j∈[0, L-1]; in, This represents the result of centering the x-coordinates of the geometric center points in the effective center point set. This represents the result of centering the ordinates of the geometric center points in the effective center point set; , These represent the x and y coordinates of several center points, respectively. Step S6.2, shift the centered data to a mean of zero: ； in, Let be the relative center coordinates, representing the zero-mean matrix after the overall translation of the effective center point set; Step S6.3, calculate the covariance matrix of the translated zero-mean matrix: ； in, Represent the covariance matrix; Indicates the variance in the x-direction; = Represents covariance; Indicates the variance in the y-direction; Step S6.4, for the covariance matrix Perform eigenvalue decomposition: ； in, Represents eigenvalues. represents the eigenvector; the eigenvector corresponding to the largest eigenvalue represents the principal direction, and the eigenvector corresponding to the smallest eigenvalue represents the secondary direction. Step S6.5, calculate the mesh tilt angle: ； in, , These are the elements of the eigenvector corresponding to the largest eigenvalue.

5. The microlens array calibration and detection method according to claim 1, characterized in that, Step S7 specifically includes: Step S7.1, for each point ( Find its nearest neighbor and calculate the Euclidean distance. : ,i,j∈[0, L-1]; Step S7.2, based on Euclidean distance Calculate the average spacing between geometric center points : 。 6. The microlens array calibration and detection method according to claim 1, characterized in that, The threshold set in step S9 is 1.3-1.7 times the average spacing between geometric center points.

7. The microlens array calibration and detection method according to claim 1, characterized in that, In step S10, the value range of the set threshold is 95%-98%.

8. A light field image processing method based on a refocusing algorithm, characterized in that, The method includes the following steps: Step (1): The microlens array of the light field camera is calibrated using the microlens array calibration and detection method as described in any one of claims 1-7. If the calibration is qualified, the following steps are performed. Step (2): Obtain the original image containing microlens array imaging information and perform grayscale processing to obtain a grayscale image, which is expressed as follows: ； Where W and H are the width and height of the original CCD image; Step (3): The grayscale image is segmented according to the geometric parameters of the microlens array, and the sub-images corresponding to each microlens are extracted to construct the four-dimensional light field data corresponding to each microlens: ； m∈[1,M],n∈[1,N],u∈[1,U],v∈[1,V]; Where m is the row of the microlens array, n is the column of the microlens array; u is the sub-image row, v is the sub-image column; Step (4): Arrange the four-dimensional light field data according to the structure of the microlens array to generate complete light field cube data containing spatial dimensions (m,n) and angular dimensions (u,v): ； in, Indicates rearrangement; Step (5): Under a fixed angular direction (u0, v0), extract all pixels at that angle to obtain the sub-aperture image. ； Step (6): Determine the position of the virtual refocusing plane according to the lens imaging formula, and calculate L1: ； Where f is the focal length of the main lens, U1 is the original reference plane distance, U2 is the target focusing plane distance, L is the original phase distance, and L1 is the target image distance. And calculate the geometric scaling factor: ; Step (7), calculate the pixel offset of the sub-aperture image: ； Where: Δ represents the pixel offset spacing, Δ=D·p, D is the normalized scaling factor, 1<D<100; Normalized to pixel offset: ; Step (8): Offset compensation is performed on the fixed-angle view, and weighted fusion is performed to obtain the refocused image: ； in, , W is the weighting function.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the microlens array calibration and detection method as described in any one of claims 1-7, or the light field image processing method as described in claim 8.

Images