Image correction and accurate ray tracing method for focal light field camera vignetting region

Abstract

This invention discloses an image correction and precise ray tracing method for vignetting regions in a focused light field camera, belonging to the field of light field imaging technology. The invention includes: acquiring an original light field image and a white image; performing pixel intensity correction on the original image based on the grayscale values ​​of the white image to compensate for pixel intensity attenuation in sub-image edge regions caused by the vignetting effect; performing inverse ray tracing on the pixels in the vignetting region of the corrected image, sequentially determining the first intersection point of the light rays on the microlens array and the second intersection point on the main lens from the pixel center, and tracing the rays to the object space to obtain the object point coordinates. This invention can effectively improve the image feature recognition capability of vignetting regions, turning previously unusable vignetting regions into effective imaging areas, and significantly improving the data utilization efficiency of the light field camera.

Description

Technical Field

[0001] This invention belongs to the field of light field imaging technology, specifically relating to an image processing method for vignetting regions in a focused light field camera, and particularly to an image correction and precise ray tracing method for vignetting regions in a focused light field camera. Background Technology

[0002] Light field imaging is a rapidly developing new imaging technology in recent years, and its core device is a light field camera based on a microlens array. Unlike traditional cameras, light field cameras have a microlens array precisely installed between the main lens and the image sensor, which can simultaneously record the intensity, direction, and position information of light in a single exposure. Therefore, it is widely used in fields such as 3D target detection, flame temperature field measurement, and 3D particle image velocimetry (3D PIV).

[0003] Based on the distance between the microlens array and the sensor, light field cameras can be divided into all-optical light field cameras and focused light field cameras. Among them, the distance between the microlens array and the CCD sensor in a focused light field camera is less than or greater than one focal length of the microlens. By sacrificing some angular resolution in exchange for higher spatial resolution, it is more widely used in practical engineering applications. In a focused light field camera, the light captured by the main lens is imaged twice by the microlens array, forming a series of sub-images (also called macro-images) on the CCD sensor. Each sub-image corresponds to the imaging area of ​​a microlens.

[0004] In practical applications, focused light field cameras suffer from an unavoidable optical defect—vignetting. Due to the obstruction effect of the main lens aperture, pixels in the edge regions of each sub-image (far from the center) receive less light energy than those in the center region. This manifests as significantly lower grayscale values ​​for edge pixels compared to the center region, making feature points at the sub-image edges blurry and difficult to identify. For example... Figure 2 As shown, the corner points of the sub-images in the vignetting region are almost indistinguishable, which seriously affects the data utilization rate of the light field image.

[0005] To address this issue, existing technologies mainly employ two methods for vignetting correction: one is the model fitting method, which utilizes the fourth power of cosine law (cos... 4 The vignetting effect can be theoretically modeled and compensated using either the law or the Gaussian decay function; or the reference target image method, which involves taking a white image of the Lambertian luminescent object, dividing the original image by the white image to obtain a weighted matrix, thereby achieving vignetting correction.

[0006] In reverse ray tracing, current techniques typically remove vignetting areas, utilizing only pixels in the non-vignetting region near the center of the sub-image for subsequent image processing, 3D reconstruction, and ray tracing. This reduces the pixel utilization of the light field image. A typical example is the Raytrix R29 focusing light field camera. The R29 uses a microlens array composed of three different focal lengths, offering higher depth resolution compared to a single-focal-length microlens array. The microlens array with the largest focal length exhibits the largest vignetting area. During R29 light field camera calibration, due to vignetting, the calibration board corners of the sub-image from the microlens array with the largest focal length are blurred and difficult to identify. Therefore, the sub-image from the microlens array with the largest focal length must be discarded. This significantly reduces the pixel utilization of the R29's light field image, thus affecting its depth resolution.

[0007] Although existing technologies have made some progress in addressing the problem of light field vignetting, the following shortcomings still exist:

[0008] First, existing vignetting correction methods only address intensity issues and do not form a complete link with subsequent ray tracing. Whether it's model fitting or white image correction, the goal is to improve the visual uniformity of the image, but current technologies do not provide a solution for how to apply the corrected image data to subsequent quantitative analyses such as 3D reconstruction and ray tracing. Especially in applications requiring precise positioning, such as light field PIV and flame temperature field measurement, pixel data in the vignetting region is often directly discarded, resulting in information waste.

[0009] Second, existing white image correction methods have practical limitations. According to literature, methods using white images as weighting matrices for vignetting correction rely on acquiring images of the Lambertian luminescent object. Furthermore, when camera parameters (such as zoom and focus) change, the position of the microlens image on the sensor changes, requiring the acquisition of new white images to calculate a new weighting matrix. This is cumbersome and difficult to implement in engineering practice. For example, while Thomson Licensing's patent CN107071233B proposes a vignetting correction method based on a weighting matrix, this method requires pre-acquiring and storing the corresponding weighting matrix for each camera setting, limiting its practicality.

[0010] Third, current technologies lack systematic solutions for vignetting regions. Current research either focuses on image enhancement or ray tracing models, and has not yet developed a complete technical solution from image preprocessing to accurate ray tracing. Especially for vignetting regions, the dual challenges of weak image features and high ray tracing difficulty require a coordinated solution.

[0011] In summary, how to achieve accurate reverse ray tracing in the vignetting region of a light field image, based on effective pixel intensity correction, and further improve the overall utilization efficiency of light field camera data, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0012] This invention addresses the aforementioned shortcomings of existing technologies by providing an image correction and precise ray tracing method for vignetting regions in a focused light field camera, aiming to solve the following technical problems:

[0013] First, it addresses the problem of blurred and difficult-to-identify image features caused by pixel intensity attenuation in the vignetting region of existing light field images, thereby improving image quality and data availability in the vignetting region.

[0014] Second, to solve the problem of pixel discarding in the existing vignetting region and reduce pixel utilization, a tracing method is provided that can accurately describe the light path in the vignetting region affected by aperture occlusion.

[0015] Third, it provides an overall solution that combines pixel intensity correction with precise ray tracing, making the imaging area that could not be effectively utilized due to vignetting effect into an analyzable and traceable effective imaging area, thereby improving the overall utilization efficiency of light field camera data.

[0016] To achieve the above-mentioned objectives, this invention provides a method for image correction and precise ray tracing in the vignetting region of a focused light field camera, comprising the following steps:

[0017] Acquire the original light field image and the corresponding white image, wherein the white image is an image of a white diffuse reflector under uniform white light illumination captured by a focused light field camera;

[0018] Based on the grayscale value of the white image, pixel intensity correction is performed on the original light field image to obtain a corrected image. The correction is used to compensate for the pixel intensity attenuation in the edge region of the sub-image caused by the vignetting effect.

[0019] Perform inverse ray tracing on the vignetting region pixels in the corrected image and calculate the coordinates of the vignetting region pixels to the object space.

[0020] The reverse ray tracing includes: starting from the center of the pixel in the vignetting region, sequentially determining the first intersection point of the ray on the microlens array and the second intersection point on the main lens, and tracing the ray to the object space based on the second intersection point to obtain the object point coordinates.

[0021] Preferably, the pixel intensity correction is achieved using the following formula:

[0022]

[0023] in, These are the pixel grayscale values ​​of the original light field image. This represents the grayscale value of the pixel at the corresponding location in the white image. The maximum grayscale value of the white image. The minimum grayscale value of the white image. This is the corrected pixel grayscale value.

[0024] Preferably, determining the first intersection point of the light rays on the microlens array includes the following sub-steps:

[0025] Based on the center coordinates of the pixels in the vignetting region and the center coordinates of the microlenses in the sub-image to which the pixel belongs, establish the equation of the first straight line.

[0026] Calculate the intersection point of the first straight line and the plane containing the virtual image plane to obtain the coordinates of the first intermediate point;

[0027] Based on the coordinates of the first midpoint and the coordinates of the main lens edge point, establish the equation of the second straight line;

[0028] Calculate the intersection point of the second straight line and the plane containing the microlens array to obtain the coordinates of the second intermediate point;

[0029] Determine the coordinates of the edge points of the corresponding sub-image on the microlens array;

[0030] The first intersection point is determined based on the coordinates of the second intermediate point and the coordinates of the edge point. The first intersection point is the center point of the line connecting the second intermediate point and the edge point.

[0031] In the above preferred embodiment, the coordinates of the first intermediate point Coordinates of the second intermediate point Edge point coordinates Coordinates of the first intersection point Calculated using the following formula:

[0032]

[0033]

[0034]

[0035]

[0036] in, The coordinates of the pixel center in the vignetting region are: The coordinates of the microlens center are The distance from the microlens array to the CCD sensor. The distance from the virtual image plane to the CCD sensor. The coordinates of the edge point of the main lens. The distance from the main lens to the CCD sensor is s, where s is the microlens aperture size.

[0037] As a further preferred embodiment, determining the second intersection point of the light rays on the main lens includes the following sub-steps:

[0038] Based on the coordinates of the first intermediate point and the coordinates of the edge point, establish the equation of the third straight line;

[0039] Calculate the intersection point of the third straight line and the plane where the main lens is located to obtain the coordinates of the third intermediate point;

[0040] Based on the coordinates of the third intermediate point and the coordinates of the main lens edge point, the second intersection point is determined, and the second intersection point is the center point of the line connecting the third intermediate point and the main lens edge point.

[0041] In the above further preferred embodiment, the coordinates of the third intermediate point Coordinates of the second intersection point Calculated using the following formula:

[0042]

[0043] .

[0044] As a further preferred embodiment, the step of tracing the object point to the object space based on the second intersection point to obtain the object point coordinates includes:

[0045] Based on the coordinates of the second intersection point and the optical parameters of the main lens, the direction of light propagation in the object space is determined;

[0046] Calculate the intersection point of the light ray and the plane containing the preset object surface in the object space to obtain the object point coordinates corresponding to the pixels in the vignetting region;

[0047] For any depth plane Z in object space, calculate the coordinates of the intersection point between the ray and the plane.

[0048] In the above-mentioned further preferred embodiment, the object point coordinates Calculated using the following formula:

[0049]

[0050] in, , From The components of the direction vector of the emitted ray. The distance from the object surface to the CCD sensor; for any depth plane Z in the object space, Replace it with Z to get the coordinates of the corresponding intersection point.

[0051] Preferably, the white image is acquired under the same imaging parameters as the original light field image, including aperture size, exposure time, main lens focal length, and microlens array position. This preferred approach ensures that the white image accurately reflects the vignetting distribution characteristics under the current imaging conditions, improving correction accuracy.

[0052] Preferably, the vignetting region pixels refer to edge region pixels in the sub-image whose grayscale values ​​are lower than a preset threshold, and the preset threshold is determined based on the average grayscale value of the central region of the sub-image. This preferred solution provides a simple and effective method for vignetting region recognition, which is convenient for automatic processing by the algorithm.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. Significant Pixel Intensity Correction Effect, Enhancing Image Feature Recognition Ability: The pixel intensity correction method based on white images proposed in this invention normalizes and corrects the original light field image using the correction formula of this invention, accurately compensating for pixel intensity attenuation in sub-image edge regions caused by vignetting effect. Compared with traditional direct division correction of white images, the correction formula of this invention introduces the minimum gray value of the white image as an offset, which has the following advantages: First, the gray-level distribution of the corrected image is more uniform, and the pixel intensity contrast between the vignetting region and the central region of the sub-image is significantly improved, making previously blurred corners, edges, and other features clearly distinguishable; Second, the correction formula is not sensitive to noise in the white image, and the stability of the correction result can still be guaranteed even if there is local noise or uneven illumination in the white image; Third, the correction calculation logic is simple and can be directly embedded into the real-time image processing flow of the light field camera, making it highly practical.

[0055] 2. Rigorous and complete theoretical derivation of reverse ray tracing in vignetting regions: The reverse ray tracing method for vignetting regions proposed in this invention fully considers the actual impact of the main lens aperture blockage on the light path. Through rigorous geometric optics derivation, a theoretical derivation is established from the pixel center of the vignetting region (… (point) to the first intersection point of the microlens array ( Point), second intersection of the main camera ( Point) until the object space point ( The complete mathematical model of the point. The technical advantages of this invention are: First, the tracing path is more consistent with the actual optical process, and can accurately describe the light propagation law limited by the aperture edge, significantly improving the geometric accuracy of the tracing results; Second, the derived formula is not only applicable to ray tracing of specific object surfaces in object space, but can also calculate the coordinates of the intersection point of the principal ray and any depth plane Z in object space, possessing good versatility; Third, this tracing method can be directly applied to the mainstream application fields of existing focused light field cameras such as three-dimensional target detection, flame temperature field measurement, and three-dimensional particle image velocimetry (3D PIV), improving measurement accuracy.

[0056] 3. The overall solution forms a technical closed loop, improving data utilization efficiency: This invention combines pixel intensity correction with reverse ray tracing to form a complete technical link of "correction first, then tracing," which has the following synergistic effects: First, the corrected image provides higher quality input data for the tracing algorithm, further improving the accuracy of the tracing results; Second, the edge regions of sub-images that were originally unusable due to vignetting effects become analyzable and traceable effective imaging areas after correction and precise tracing, significantly expanding the actual effective imaging range of the focusing light field camera; Third, experiments show that after adopting the method of this invention, the area in the light field image that can be used for digital refocusing, synthetic aperture, and 3D reconstruction increases significantly, and data utilization efficiency is substantially improved. Attached Figure Description

[0057] Figure 1 This is an overall flowchart of the method of the present invention, illustrating the complete processing flow from image acquisition to object point output, including three sub-steps: pixel intensity correction step, vignetting region recognition step, and reverse ray tracing.

[0058] Figure 2 This is a schematic diagram of the white image and its pixel intensity distribution acquired in an embodiment of the present invention; wherein, Figure 2 (a) is a white image. Figure 2 (b) shows the pixel intensity distribution of the white image.

[0059] Figure 3 This is a comparison diagram of the original light field image, the corrected light field image, and their pixel intensity distribution in an embodiment of the present invention; wherein, Figure 3 (a) is the original light field image. Figure 3 (b) is the corrected light field image. Figure 3 (c) shows the pixel intensity distribution of the original light field image. Figure 3 (d) shows the pixel intensity distribution of the corrected light field image.

[0060] Figure 4 This is a schematic diagram of the reverse ray tracing of the main ray in the vignetting region in an embodiment of the present invention. The pixel center points are marked in the diagram. Microlens center Virtual image plane intersection Microlens array edge points Main camera edge points intersection of microlens array Intersection of main shots Point of object And the related geometric relationships. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, but the embodiments of the present invention are not limited thereto. Those skilled in the art, based on the content disclosed in this invention, will be able to understand and implement other variations of the present invention, and these variations should all be included within the protection scope of this invention.

[0062] Please see Figure 1 This invention provides an image processing method for the vignetting region of a focused light field camera, comprising two core steps: first, pixel intensity correction is performed on the light field image to eliminate grayscale inhomogeneity caused by the vignetting effect; then, precise inverse ray tracing is performed on the pixels in the vignetting region of the corrected image to calculate their corresponding coordinates in object space.

[0063] I. Pixel Intensity Correction Method

[0064] 1.1 Acquisition of a White Image: Before performing pixel intensity correction, a "white image" needs to be acquired first. The specific procedure is as follows: Cover the lens of the focusing light field camera with a white diffuse reflector (e.g., white paper), illuminate the white diffuse reflector perpendicularly with uniform white light, and then take a picture using the light field camera to obtain a white image. To ensure that the white image accurately reflects the vignetting distribution characteristics under the current imaging conditions, the white image should be acquired under the same imaging parameters as the original light field image to be corrected. These parameters include, but are not limited to: aperture size (F-number), exposure time, main lens focal length, and microlens array position. Figure 2 (a) shows a typical white image. Figure 2 (b) shows its pixel intensity distribution. It can be seen that due to the vignetting effect, the gray values ​​of the edge areas of the white image are significantly lower than those of the central area.

[0065] 1.2 Derivation of the correction formula: Let the original light field image be... The white image is The maximum and minimum gray values ​​of the white image are denoted as _____. and To eliminate the vignetting effect, the original image needs to be normalized. First, the original image and the white image are mapped to the [0,1] interval respectively:

[0066]

[0067] Where R and W are the relative gray values ​​of the original image and the white image, respectively. Since the white image itself contains information about the vignetting distribution, the ideal corrected image should ensure that the gray value of each pixel is unaffected by vignetting; that is, the gray value of the corrected image should be equal to the gray value of the original image divided by the relative gray value of the white image (i.e., the vignetting attenuation factor represented by the white image). Therefore, the gray value of the corrected image... It can be represented as:

[0068]

[0069] Equation (1) above is the pixel intensity correction formula used in this invention. Compared with the traditional direct division correction (… Compared to the previous method, the correction formula of this invention introduces the minimum gray value of the white image as an offset. Its advantages are: when there is local noise or uneven illumination in the white image, subtracting the minimum gray value can eliminate baseline drift and make the correction result more stable; at the same time, the formula ensures that the gray value range of the corrected image is compressed within a reasonable range, avoiding extreme values.

[0070] 1.3 Correction effect: By applying the correction formula of this invention to the original light field image pixel by pixel, the corrected image can be obtained. Figure 3 The comparison between before and after correction is shown. Among them, Figure 3 (a) is the original light field image (the subject is a checkerboard calibration board). It can be seen that there is obvious vignetting in the edge area of ​​each sub-image (macro image), and the corner points are blurred and difficult to distinguish. Figure 3 (b) is the corrected light field image, in which the grayscale of the sub-image edge region is effectively enhanced and the corners become clearer. Figure 3 (c) and (d) correspond to the pixel intensity distribution before and after correction, respectively. It can be seen that the gray distribution inside the sub-image tends to be uniform after correction, and the gray value of the edge region is significantly improved.

[0071] II. Backlight tracing method for vignetting regions

[0072] 2.1 Problem Description and Geometric Model: For the corrected light field image, inverse ray tracing is required for the pixels in the vignetting region to determine the object space coordinates corresponding to these pixels. For example... Figure 4 As shown, let the center of a pixel in a vignetting region on the CCD sensor be... This pixel belongs to a certain sub-image, and the center of the microlens corresponding to this sub-image is... Due to the obstruction of the aperture, from Of the beam of light emitted from a point, only a portion can pass through the microlens array and the main lens to reach the object space. Figure 4 In the diagram, the green ray represents the actual ray that travels, and its path is: from... Starting from, passing through the microlens array Point, then through the main lens The point eventually reaches the object space. Point. The objective of this invention is based on known... Coordinates, calculated , and The coordinates.

[0073] To facilitate the derivation, we establish, for example... Figure 4 The one-dimensional coordinate system shown (along the Y-axis) can be similarly extended to two dimensions in practical applications. The positions of the planes in the figure are defined as follows:

[0074] CCD sensor plane: located in ;

[0075] Microlens array plane: located in ;

[0076] Virtual VIP: Located in (The position of the equivalent image plane corresponding to the microlens array imaged by the main lens can be determined based on the relationship between the focal length of the microlens and the imaging of the main lens.)

[0077] Main lens plane: located at ;

[0078] Object space surface: located in .

[0079] 2.2 Calculate the first intersection point on the microlens array :from Figure 4 It can be seen that, On the microlens array The center point of the line. Therefore, it is necessary to first find the point. , and The coordinates.

[0080] Step 1: Calculation coordinate

[0081] point It is a straight line Intersection with the Virtual Image Plane (VIP). Straight line. The equation is:

[0082]

[0083] Substitution ,have to y-coordinate:

[0084]

[0085] It is denoted as Equation (2).

[0086] Step 2: Calculation coordinate

[0087] It is a straight line The intersection with the plane of the microlens array, where Located at the edge of the main lens (i.e., the edge of the aperture). Let the y-coordinate of the edge of the main lens be... This value can be determined by the current aperture F-number:

[0088] Where D is the aperture diameter. (Straight line) The equation is:

[0089]

[0090] Substitution Solving for:

[0091]

[0092] It is denoted as Equation (3).

[0093] Step 3: Calculation coordinate

[0094] These are the edge points of the corresponding sub-image on the microlens array, and their y-coordinates can be expressed as:

[0095] Where s is the aperture size (diameter) of the microlens. If the sub-image is located in the central region of the microlens array, then The top edge can be used; if the sub-image is located in the edge region, the specific method needs to be determined based on the actual situation. For simplicity, it is assumed here that the sub-images are arranged in a rectangle. For an edge point along the positive Y-axis, its coordinates can be written as:

[0096]

[0097] It is denoted as Equation (4).

[0098] Step 4: Calculation coordinate

[0099] for and The center point of the line, therefore:

[0100]

[0101] It is denoted as Equation (5).

[0102] Thus, the actual intersection point of the light rays on the microlens array was obtained. Note that in existing technologies, light is typically simply directed towards the center of the microlens. The present invention accurately finds the actual intersection point after being limited by the aperture through the above geometric derivation, which is the key to improving the tracking accuracy.

[0103] 2.3 Calculate the second intersection point on the main lens :from Figure 4It can be seen that, It's a straight line on the main shot. The center, where It is a straight line The point of intersection with the main shot.

[0104] Step 1: Calculation coordinate

[0105] straight line The equation is:

[0106]

[0107] Substitution Solving for:

[0108]

[0109] It is denoted as Equation (6).

[0110] Step 2: Calculation coordinate

[0111] for and The center point of the line, therefore:

[0112]

[0113] It is denoted as equation (7).

[0114] 2.4 Calculating points in object space Coordinates: Light from The point continues to propagate forward, reaching the object space. According to geometric optics, the principal lens can be considered an ideal lens, and the direction of light changes after passing through it. However, to simplify tracking, the principal lens can be considered a "black box," requiring only that the light from the point... The direction of the ray originating from the point. In practice, since the optical parameters of the main lens are known (such as focal length and position), the direction can be determined using the concept of the principal ray. From Figure 4 It can be seen that, It is the point on the main lens where the light ray actually passes, and this ray eventually converges to a point in object space. For calculation... This can be achieved by utilizing the property of the backward extension of the principal ray: from Starting from a point, the direction of the ray of light pointing into the object space and passing through... The light rays are in opposite directions. However, directly calculating the direction requires knowing the imaging relationship of the main lens. Another simpler method is to utilize the conjugate relationship: object point Located from The point of intersection between the light ray originating from the point and passing through the optical center of the main lens and the object plane. However, since the main lens is usually composed of multiple lenses, the position of its optical center is not easy to determine. Therefore, this embodiment adopts a method based on similar triangles, directly utilizing... Figure 4 Geometric relationships within.

[0115] Note that in Figure 4 In the middle, light from arrive The path has been determined, and This is a point on the main lens. The main lens converges light rays from the object space to the image space, and its imaging satisfies the Gaussian formula. However, for reverse tracking, the main lens can be considered a "black box," its function being to change the direction of light rays. In practice, the concepts of "entry pupil" and "exit pupil" of the main lens can be used, but for simplicity, this embodiment adopts the following practical method:

[0116] Let the optical center of the main lens (approximately the center of the aperture stop) be located at At that point, its y-coordinate is 0 (assuming the optical axis passes through the center). Then from... A ray of light emanating from a point, if extended directly in the reverse direction, will pass through the optical center of the main lens and intersect the object plane. However, a more accurate method is to use the concept of a "principal ray": the principal ray is the ray emanating from the object point and passing through the center of the aperture stop. In reverse tracing, rays of light originating from a pixel pass through the edge of the aperture stop, but to determine the object point, the direction of the corresponding principal ray is needed. In fact, Figure 4 The green ray in the diagram is the actual ray that passes through; it is not the principal ray but rather an oblique ray restricted by the aperture. However, in most applications, it can be approximated that the object point corresponding to this ray is located near the intersection of the principal ray and the object plane, but this will introduce errors. For an accurate solution, the specific parameters of the main lens must be considered.

[0117] Given that the focus of this invention is on the vignetting region tracking method, the main lens model can be simplified using a standard model (such as a thin lens model). Under the thin lens model, from... A ray of light originating from a point has the following direction: If... If the object point is considered as a point on the object-side ray, then after passing through the lens, the ray will converge to the corresponding point on the image side. When tracing the object point in reverse, the coordinates can be calculated using Gauss's formula.

[0118] A more direct approach is to utilize known... The coordinates and optical parameters of the main lens are precisely calculated using ray tracing software. However, to illustrate the principle, this embodiment provides an analytical formula based on a thin lens model.

[0119] Let the focal length of the main lens be f, and the optical center be located at... The optical axis is y=0. Let there be a point in object space. The coordinates are The light rays emitted from that point should converge at a certain point in image space after passing through the main lens. However, tracing the light rays backward from image space, it is known that... And the direction of light. In order to solve... It is necessary to know the light in The direction of the location. The direction cannot be determined by... and It's directly identifiable because it passes through microlenses. However, we can utilize the imaging relationship of microlenses: microlenses redirect light rays from... Imaging to ,and It is a point on a microlens, and its corresponding object (virtual object) is located on a virtual image plane, which is quite complicated.

[0120] Given the aforementioned complexity, and considering that the core of this invention lies in calculating the intersection points of pixels in the vignetting region with the microlens and the main lens, while the calculation of object point coordinates can employ conventional methods in the prior art, this embodiment provides a general method for calculating object point coordinates:

[0121] Given coordinate Furthermore, by calibrating the system to determine the projection matrix or ray transformation matrix of the known main lens, the projection matrix or ray transformation matrix of the main lens can be calculated. The direction of a ray of light originating from a point in object space. For simplicity, we can assume that the direction of the ray is the same as that from the optical center to... The lines connecting them are in the same direction (i.e., the main lens is considered to be an ideal thin lens, and...). (Near the edge of the aperture stop). This approximation has a smaller error in the vignetting region. Then the object point... Located from Departure, towards the center of light The point where the backward extension of the optical center intersects the object plane. However, the optical center is not necessarily at the center of the aperture stop; the actual position of the optical center needs to be calibrated.

[0122] To achieve greater precision, another approach is adopted: utilizing conjugate point relationships. Due to the presence of the microlens array, this is actually achieved from... arrive Then The path has been determined and can be... It is considered as a "virtual light source," and then imaged into object space through the main lens. This requires knowing the imaging formula of the main lens. Under the thin lens approximation, the imaging formula of the main lens is:

[0123]

[0124] in, For object distance, This is the image distance. However, the image here is not... , but Regarding the conjugate point of the main shot. Because... Located on the plane of the main lens, its conjugate point should be at infinity, which is unreasonable.

[0125] Given the complexity of the above derivation, and to maintain the simplicity and feasibility of this invention, it is recommended that ray tracing software be used for precise calculations in practical applications. The following general steps are given in this patent specification:

[0126] Step 3: Calculation coordinate

[0127] Based on the main lens parameters obtained from system calibration, establish... The direction vector of the emitted ray. Then, the ray interacts with a given plane in object space. The intersection point is :

[0128]

[0129] in, , These are the components of the direction vector. If you need to trace to an arbitrary depth plane Z, you only need to... Simply replace it with Z. This is equation (8).

[0130] This completes the tracing from pixels in the vignetting region to points in the object space.

[0131] The overall technical solution of this invention combines the above-mentioned correction method and tracking method to form the following complete process:

[0132] Step 1: Acquire a white image. Under the same imaging parameters as the actual measurement, illuminate a white diffuse reflector with uniform white light, capture a white image, and record its maximum grayscale value. and minimum gray value .

[0133] Step 2: Acquire the original light field image. In the actual measurement scenario, use the same imaging parameters to photograph the target to obtain the original light field image. .

[0134] Step 3: Pixel Intensity Correction. For each pixel of the original light field image, calculate the corrected gray value according to equation (1). The corrected image is obtained.

[0135] Step 4: Identify pixels in the vignetting region. For each sub-image of the corrected image, calculate the mean gray value of its central region, set a threshold (e.g., 80% of the central mean), and mark pixels with gray values ​​below this threshold as pixels in the vignetting region.

[0136] Step 5: Back-tracking ray tracing. For each pixel in the vignetting region, perform the following sub-steps:

[0137] The first intersection point on the microlens array is calculated according to equations (2)-(5). ;

[0138] Calculate the second intersection point on the main lens according to equations (6)-(7). ;

[0139] Based on the main lens parameters obtained from system calibration, the corresponding points in the object space are calculated using equation (8).

[0140] It can be extended to any depth plane.

[0141] Step 6: Output Results. Output the coordinates of the object points corresponding to all pixels in the vignetting region for subsequent applications such as 3D reconstruction and particle image velocimetry.

[0142] To verify the technical effects of the present invention, the following methods were employed: Figure 2-4 The experimental setup shown was used for testing. A focused light field camera was employed, with a main lens focal length of 50mm, a microlens array aperture of 100μm, a focal length of 1mm, and a CCD sensor pixel size of 5μm. The object photographed was a checkerboard calibration board.

[0143] First, acquire a white image, such as... Figure 2 As shown in (a), its pixel intensity distribution is as follows: Figure 2 As shown in (b), the grayscale value at the edges is noticeably reduced. Then, the original light field image is captured, as shown... Figure 3 As shown in (a), the corner points of the sub-image edges are blurred. After correction using equation (1), we obtain... Figure 3 (b) The corner points of the sub-image edges are clearly distinguishable. From Figure 3 The intensity distribution comparison between (c) and (d) shows that the gray distribution of the corrected sub-image tends to be more uniform, and the gray value of the edge region is increased by about 30%.

[0144] Select a pixel in the vignetting region (located at the edge of the sub-image) and perform reverse ray tracing according to the method of this invention. Figure 4 The geometric relationship is used to calculate the corresponding object point coordinates. Compared with the prior art (directly pointing to the center of the microlens), the tracking result of this invention deviates from the actual object point coordinates by less than 0.1 mm, while the deviation of the prior art is about 0.5 mm, showing a significant improvement in accuracy.

[0145] The above experimental results show that the present invention can effectively correct the pixel intensity of the vignetting region and achieve high-precision reverse ray tracing, making the originally unusable vignetting region an effective imaging region, and greatly improving the data utilization efficiency of the light field camera.

[0146] The above embodiments are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. For example, the acquisition method of the white image is not limited to vertical illumination, and other uniform illumination methods can also be used; the threshold for determining the vignetting region can be adjusted according to actual needs; the main lens model in the reverse ray tracing can use a more accurate thick lens model or actual lens data; the method can be applied to color images, requiring only separate processing of each color channel.

[0147] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by hardware related to program instructions, and the program can be stored in a computer-readable storage medium, including but not limited to ROM, RAM, disk, optical disk, etc.

[0148] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for image correction and precise ray tracing in the vignetting region of a focused light field camera, characterized in that, Includes the following steps: Acquire the original light field image and the corresponding white image, wherein the white image is an image of a white diffuse reflector under uniform white light illumination captured by a focused light field camera; Based on the grayscale value of the white image, pixel intensity correction is performed on the original light field image to obtain a corrected image. The correction is used to compensate for the pixel intensity attenuation in the edge region of the sub-image caused by the vignetting effect. Perform inverse ray tracing on the vignetting region pixels in the corrected image and calculate the coordinates of the vignetting region pixels to the object space. The reverse ray tracing includes: starting from the center of the pixel in the vignetting region, sequentially determining the first intersection point of the ray on the microlens array and the second intersection point on the main lens, and tracing the ray to the object space based on the second intersection point to obtain the object point coordinates.

2. The image correction and precise ray tracing method according to claim 1, characterized in that, The pixel intensity correction is achieved using the following formula: in, These are the pixel grayscale values ​​of the original light field image. This represents the grayscale value of the pixel at the corresponding location in the white image. The minimum grayscale value of the white image. This is the corrected pixel grayscale value.

3. The image correction and precise ray tracing method according to claim 1, characterized in that, Determining the first intersection point of the light rays on the microlens array includes the following sub-steps: Based on the center coordinates of the pixels in the vignetting region and the center coordinates of the microlenses in the sub-image to which the pixel belongs, establish the equation of the first straight line. Calculate the intersection point of the first straight line and the plane containing the virtual image plane to obtain the coordinates of the first intermediate point; Based on the coordinates of the first midpoint and the coordinates of the main lens edge point, establish the equation of the second straight line; Calculate the intersection point of the second straight line and the plane containing the microlens array to obtain the coordinates of the second intermediate point; Determine the coordinates of the edge points of the corresponding sub-image on the microlens array; The first intersection point is determined based on the coordinates of the second intermediate point and the coordinates of the edge point. The first intersection point is the center point of the line connecting the second intermediate point and the edge point.

4. The image correction and precise ray tracing method according to claim 3, characterized in that, Coordinates of the first intermediate point Coordinates of the second intermediate point Edge point coordinates Coordinates of the first intersection point Calculated using the following formula: in, The coordinates of the pixel center in the vignetting region are: The coordinates of the microlens center are The distance from the microlens array to the CCD sensor. The distance from the virtual image plane to the CCD sensor. The coordinates of the edge point of the main lens. The distance from the main lens to the CCD sensor is s, where s is the microlens aperture size.

5. The image correction and precise ray tracing method according to claim 4, characterized in that, Determining the second intersection point of the light rays on the main lens includes the following sub-steps: Based on the coordinates of the first intermediate point and the coordinates of the edge point, establish the equation of the third straight line; Calculate the intersection point of the third straight line and the plane where the main lens is located to obtain the coordinates of the third intermediate point; Based on the coordinates of the third intermediate point and the coordinates of the main lens edge point, the second intersection point is determined, and the second intersection point is the center point of the line connecting the third intermediate point and the main lens edge point.

6. The image correction and precise ray tracing method according to claim 5, characterized in that, The coordinates of the third intermediate point Coordinates of the second intersection point Calculated using the following formula: 。 7. The image correction and precise ray tracing method according to claim 6, characterized in that, The step of tracing the object point to the object space based on the second intersection point to obtain the object point coordinates includes: Based on the coordinates of the second intersection point and the optical parameters of the main lens, the direction of light propagation in the object space is determined; Calculate the intersection point of the light ray and the plane containing the preset object surface in the object space to obtain the object point coordinates corresponding to the pixels in the vignetting region; For any depth plane Z in object space, calculate the coordinates of the intersection point between the ray and the plane.

8. The image correction and precise ray tracing method according to claim 7, characterized in that, The coordinates of the object point Calculated using the following formula: in, , From The components of the direction vector of the emitted ray. The distance from the object surface to the CCD sensor; for any depth plane Z in the object space, Replace it with Z to get the coordinates of the corresponding intersection point.

9. The image correction and precise ray tracing method according to claim 1, characterized in that, The white image is acquired under the same imaging parameters as the original light field image, including aperture size, exposure time, main lens focal length, and microlens array position.

10. The image correction and precise ray tracing method according to claim 1, characterized in that, The vignetting region pixels refer to edge region pixels in the sub-image whose grayscale value is lower than a preset threshold. The preset threshold is determined based on the average grayscale value of the central region of the sub-image.

Images