Light Field Camera Calibration Method Based on Line Feature Detection of Original Light Field Images

Abstract

The present invention relates to the field of light field imaging, and particularly to a light field camera calibration method based on line feature detection of an original light field image, including obtaining an original light field image set of a white background of a light field camera under different shooting conditions, and processing to obtain projection center grid data of a microlens array; obtaining an original light field image set of a checkerboard of a light field camera under different shooting conditions, and loading the projection center grid data of the microlens array on each original light field image of the checkerboard; calculating the gradient value of each pixel in each original light field image of the checkerboard, queuing the gradient values of the pixels, performing regional diffusion and model fitting in the direction of similar gradient angles, obtaining line features in each original light field image of the checkerboard, and thus calculating the corner coordinates of each original light field image of the checkerboard; constructing a light field camera projection model, and combining the foregoing corner coordinates to calibrate the parameters of the light field camera. The present invention saves time and reduces the introduced error at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of light field imaging, and particularly relates to a light field camera calibration method based on line feature detection of an original light field image. Background Art

[0002] As a new imaging method, light field imaging technology can capture the three-dimensional spatial distribution information of light in a scene, providing a new research direction for the field of computer vision. However, due to the high-dimensional characteristics of the original light field image (including spatial information and angular information), traditional two-dimensional image-based calibration methods are difficult to be directly applied to the calibration of light field cameras.

[0003] The traditional calibration method of a light field camera is to perform preprocessing such as de-vignetting, rotation, translation, demosaicing, resampling, interpolation, etc. on the original light field image to obtain sub-aperture images, and then perform corner detection calibration on the sub-aperture images; due to the cumbersome process and the lack of direct calibration of the original light field image, it is time-consuming and prone to errors.

[0004] Currently, there are some line feature detection algorithms for original light field images based on template matching. Such algorithms detect lines or features in the original light field image by matching with pre-defined templates. However, it is necessary to manually design pre-defined templates according to the structure of the microarray light field camera and the light field camera; for large-sized original light field images or multi-template matching, there are problems of high overhead and low computational efficiency; there are also problems of being sensitive to noise and poor adaptability to scale and rotation changes. Especially when the background in the original light field image is complex or the difference between the target object and the background is not obvious, template matching may result in incorrect matching. Summary of the Invention

[0005] In view of this, the present invention aims to provide a light field camera calibration method based on line feature detection of an original light field image, including the following steps:

[0006] S1: Obtain a set of original light field images with a white background taken by a light field camera at different focal lengths and different brightnesses, and process the set of original light field images with a white background to obtain the projection center grid data of the microlens array;

[0007] S2: Obtain a set of original light field images of a checkerboard taken by a light field camera at different focal lengths, different angles, and different brightnesses;

[0008] S3: Pair the set of original light field images of the checkerboard with the set of original light field images with a white background, and load the projection center grid data of the microlens array on each original light field image of the checkerboard;

[0009] S4: For each checkerboard original light field image loaded with the projection center grid data of the microlens array, calculate the gradient value of each pixel in the checkerboard original light field image, queue the gradient values of the pixels, select seed points, perform regional diffusion and model fitting in the direction of similar gradient angles, and obtain the line features in each checkerboard original light field image;

[0010] S5: According to the line features in each checkerboard original light field image, calculate the corner coordinates of each checkerboard original light field image;

[0011] S6: Establish the mapping relationship between the pixel coordinate system and the world coordinate system, and calibrate the parameters of the light field camera in combination with the corner coordinates of each checkerboard original light field image.

[0012] Further, in step S1, generate the initial projection center grid coordinates of the microlens array according to the white background original light field image, update the projection center position of the microlens array using the interpolation calculation and weighted average method, and generate the projection center grid data of the microlens array.

[0013] Further, in step S4, the calculation formula for the gradient value of the pixel is as follows:

[0014]

[0015]

[0016] where is the two-dimensional image of the checkerboard original light field image, is the gradient value of the pixel, is the gradient direction of the pixel.

[0017] Further, in step S4, the process of queuing the gradient values of the pixels is as follows:

[0018] First, find the maximum value of the gradient values from among the numerous gradient values ;

[0019] Secondly, based on the maximum value of the gradient values calculate the rank coefficient , and perform gradient rank division on the pixels according to the rank coefficient ; among them, the calculation formula for the rank coefficient is:

[0020]

[0021] where s is the preset rank coefficient, ;

[0022] Furthermore, quantify the gradient ranks, and the gradient value of each pixel after quantization is:

[0023]

[0024] Further, after queuing the gradient values of the pixels, a gradient threshold is set , and the gradient values less than the gradient threshold are suppressed. The calculation formula of the gradient threshold is:

[0025]

[0026] where is the gradient direction angle tolerance value for region growth, is an empirical value.

[0027] Further, the process of selecting seed points for region diffusion and model fitting in the similar direction of the gradient angle to obtain the line features in each original light field image of the checkerboard is as follows:

[0028] First, select the pixels that meet the threshold requirements and have eight neighborhoods from among many pixels as seed points;

[0029] Secondly, for the similar direction of the gradient angle of the seed points, use the region growth algorithm to merge the seed points with approximately the same field direction to obtain the line support region ;

[0030]

[0031] where is the pixel in the line support region, is the gradient direction angle of the i-th pixel in the line support region;

[0032] Then, abbreviate the line support region as r . The coordinates of the pixel r in the line support region j are , and the gradient value of the pixel j is . Calculate the center point j of the minimum bounding rectangle according to the coordinates and gradient value of the pixel ; where, the calculation formula of the center point of the minimum bounding rectangle is as follows:

[0033]

[0034] Next, fit the pixels in the line support region r . The fitting formula is as follows:

[0035]

[0036]

[0037]

[0038]

[0039] Among them, is a symmetric positive semi - definite matrix; is the variance in the horizontal direction weighted by pixel gradients, is the variance in the vertical direction weighted by pixel gradients, is the covariance weighted by pixel gradients;

[0040] Finally, the error of the fitting result is estimated using the evaluation parameters of the line features. If the error meets the conditions, the fitting result is recorded as a straight line. Among them, the evaluation parameters include the angle tolerance, the maximum / minimum line length, the number of false alarms, and the line detection sensitivity.

[0041] Furthermore, in step S5, all pixels of each original light field image of the checkerboard are traversed, and the qualified straight lines are filtered out, and the coordinates of the intersection of the straight lines are calculated, which are the corner coordinates of the original light field image of the checkerboard.

[0042] Furthermore, in step S6, the formula for calibrating the parameters of the light field camera is as follows:

[0043] ;

[0044]

[0045] Among them, is the image coordinate system, is the camera coordinate system, is the world coordinate system, is the internal parameter of the light field camera, is the rotation matrix of the light field camera, is the translation vector of the light field camera, is the focal length of the light field camera in the horizontal direction, is the focal length of the light field camera in the vertical direction.

[0046] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0047] 1. The present invention can use the original light field image for calibration, avoiding the cumbersome process from the original light field image to the sub - aperture image, saving time and reducing the introduced error.

[0048] 2. Detect directly in the line feature region of the original light field image, where the image information is richer, reducing the errors that may be introduced by resampling.

[0049] 3. Compared with the traditional template matching algorithm for detecting line features of the original light field image, the present invention does not require preset templates and matching, making the calibration process more efficient.

[0050] 4. For the settings and shooting conditions of different light field cameras, the algorithm parameters of model fitting can be adjusted, making it more adaptable to the environment.

[0051] 5. The algorithm of model fitting can enhance the detection ability of regions with weak line features by adjusting the threshold, with stronger search ability.

[0052] 6. After searching the entire original light field image to determine the pixel with the maximum gradient, divide it according to levels, and then use the region growing algorithm to set parameter thresholds such as the tolerance to flexibly detect all reasonable straight lines in the original light field image without predefined templates.

[0053] 7. The implementation of the algorithm for detecting line features of the original light field image based on model fitting is relatively simple and does not require much parameter adjustment. At the same time, the parameters of the algorithm are relatively few, so it is easy to optimize in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0055] Figure 1 is a schematic flow chart of the light field camera calibration method based on detecting line features of the original light field image according to an embodiment of the present invention.

[0056] Figure 2 is a schematic diagram of the projection center grid data of the microlens array according to an embodiment of the present invention.

[0057] Figure 3 is a schematic flow chart of detecting line features of the original light field image based on model fitting according to an embodiment of the present invention.

[0058] Figure 4 is a schematic diagram of the detection result of the line feature region of the checkerboard original light field image according to an embodiment of the present invention.

[0059] Figure 5 is a schematic diagram of the corner coordinates of the checkerboard original light field image according to an embodiment of the present invention.

[0060] Figure 6It is a schematic diagram of the projection model of the light field camera according to the embodiment of the present invention. Detailed implementation manners

[0061] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention.

[0062] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0063] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0064] In the description of the present invention, it should be noted that, unless otherwise clearly defined and limited, the terms "installed", "connected", "connected" should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected, or indirectly connected through an intermediate medium, and it may be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood through specific situations.

[0065] The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.

[0066] As Figure 1 shown, the embodiment of the present invention provides a light field camera calibration method based on the detection of line features of the original light field image, including the following steps:

[0067] S1: Obtain the original light field image set of a white background captured by a light field camera at different focal lengths and different brightness levels, and process the original light field image set of the white background to obtain the projection center grid data of the microlens array.

[0068] Obtain the light field files of a pure white background captured by a light field camera at different focal lengths and different brightness levels, decode the light field files of the pure white background to obtain the original light field images of the white background, and obtain the physical parameters of the original light field images of the white background, including the focal length, white balance, and microlens array size (the microlens array size includes the size of the microlenses and the spacing between the microlenses) of the light field camera at this time. Generate the initial projection center grid coordinates of the microlens array according to the original light field images of the white background, and use the interpolation calculation and weighted average method to update the projection center position of the microlens array. As Figure 2 shown, generate a grid coordinate file, and this grid coordinate file is the projection center grid data of the microlens array in.mat format.

[0069] S2: Obtain the original light field image set of a checkerboard captured by a light field camera at different focal lengths, different angles, and different brightness levels.

[0070] Since the focal length of the light field camera is variable, there are differences in the projection center grids of the microlens arrays under different focal length settings.

[0071] S3: Pair the original light field image set of the checkerboard with the original light field image set of the white background, and load the projection center grid data of the microlens array on each original light field image of the checkerboard.

[0072] The present invention uses the load function of matlab to read the grid coordinate file.

[0073] S4: For each original light field image of the checkerboard loaded with the projection center grid data of the microlens array, calculate the gradient value of each pixel in the original light field image of the checkerboard, queue the gradient values of the pixels, select seed points and perform regional diffusion and model fitting in the direction of similar gradient angles to obtain the line features in each original light field image of the checkerboard.

[0074] The specific process of step S4 is as Figure 3 shown.

[0075] The calculation formula for the gradient value of a pixel is as follows:

[0076]

[0077]

[0078] Among them, is the two-dimensional image of the original light field image of the checkerboard, is the gradient value of the pixel, is the gradient direction of the pixel.

[0079] The process of queuing the gradient values of pixels is as follows:

[0080] First, find the maximum value of the gradient values among numerous gradient values .

[0081] Secondly, based on the maximum value of the gradient values calculate the rank coefficient , and divide the pixels into gradient ranks according to the rank coefficient ; among them, the calculation formula of the rank coefficient is:

[0082]

[0083] where s is a preset rank coefficient, .

[0084] Dividing the ranks is equivalent to classifying them into discrete data, which can reduce the complexity of data processing. Another advantage of dividing the ranks is to reduce the influence of noise on line detection, that is, to reduce the false lines caused by tiny noise being misjudged as valid edges.

[0085] Furthermore, quantize the gradient ranks, and the gradient value of each pixel after quantization is:

[0086]

[0087] Process the pixels from the highest to the lowest quantized gradient ranks, generate seed points preferentially in the significant edge regions, and reduce invalid calculations. Group pixels with similar gradient values into the same rank to ensure the consistency of gradient values during region growth and avoid splitting continuous edges due to tiny differences. Through the normalized rank values (instead of absolute gradient values), the algorithm can automatically adapt to images with different contrasts without manually adjusting the gradient threshold.

[0088] Since there are no obvious gradient changes in the smooth regions of the original checkerboard light field image, the edge detection algorithm cannot identify valid edges, and it is easily affected by noise to form false tiny gradients. Therefore, after queuing the gradient values of pixels, set a gradient threshold to suppress the gradient values less than the gradient threshold. The gradient threshold has the following calculation formula:

[0089]

[0090] where is the tolerance value of the gradient direction angle for region growth, is an empirical value, and Set corresponding values according to the actual situation.

[0091] The process of selecting seed points and performing regional diffusion and model fitting in the direction of similar gradient angles to obtain the line features in each original light field image of the checkerboard is as follows:

[0092] First, select pixels that meet the threshold requirements and have eight neighborhoods from among numerous pixels as seed points.

[0093] Secondly, for the direction of similar gradient angles of the seed points, use the region growing algorithm to merge the seed points with approximately the same direction in the field to obtain the line support region ;

[0094]

[0095] Among them, is the pixel within the line support region, is the gradient direction angle of the i-th pixel within the line support region.

[0096] Then, abbreviate the line support region as r , the coordinates of the pixel r within the line support region j are , the gradient value of the pixel j is , calculate the center point j of the minimum bounding rectangle according to the coordinates and gradient value of the pixel ; among them, the calculation formula for the center point of the minimum bounding rectangle is as follows:

[0097]

[0098] Next, fit the pixels within the line support region r , and the fitting formula is as follows:

[0099]

[0100]

[0101]

[0102]

[0103] Among them, is a symmetric positive semi-definite matrix, and its eigenvector represents the main direction of the gradient distribution within the line support region (such as the direction of a straight line or an edge); is the variance of the horizontal direction weighted by the pixel gradient, reflecting the distribution intensity of the pixels in the horizontal direction; is the variance in the vertical direction weighted by the pixel gradient, reflecting the distribution intensity of pixels in the vertical direction; is the covariance weighted by the pixel gradient, reflecting the joint distribution characteristics of pixels in the horizontal and vertical directions.

[0104] Finally, error estimation is performed on the fitting result using the evaluation parameters of the line feature. If the error meets the conditions, the fitting result is recorded as a straight line, as Figure 4 shown. If the error does not meet the conditions, the fitting result is ignored; among them, the evaluation parameters include angle tolerance, maximum / minimum line length, number of false alarms, and line detection sensitivity.

[0105] Angle tolerance: used to define the maximum angle deviation allowed between the detected straight line and the true edge or reference direction. If the angle tolerance is set larger, line segments deviating from the true angle can be accepted. If the angle tolerance is smaller, slightly inclined line segments that actually exist may be missed.

[0106] Maximum / minimum line length: Setting the minimum line length can filter out the lines generated by noise, but it may miss the detection of true short edges. If it is too long, it may cause incorrect merging of multiple line segments.

[0107] Number of false alarms: used to prove whether the setting of the number of false alarms is reasonable from the side, as an evaluation criterion.

[0108] Number of false alarms The calculation formula is as follows:

[0109]

[0110] Among them, p represents the probability that each pixel point is a straight line, P = TP / (TP + FP), TP is the true positive in the actual detection, FP is the false positive in the actual detection, k represents the number of pixels supporting the current straight line hypothesis, and S represents the total number of line segment candidates that may be generated in the algorithm.

[0111] If the NFA value of the line candidate is lower than the set threshold, it is considered a valid straight line.

[0112] Line detection sensitivity: determines the response threshold of the algorithm to the edge gradient intensity, usually related to the threshold of the edge detector (such as the low / high threshold of Canny).

[0113] The settings of the evaluation parameters should be combined with the main lens structure, microlens array structure, and sensor structure of the light field camera. Set the thresholds of the evaluation parameters according to the microlens projection in the line feature area of the original light field image of the checkerboard to ensure that the features in the line feature microlens macro-pixel area can be detected and participate in the fitting.

[0114] S5: Calculate the corner coordinates of each original light field image of the checkerboard based on the line features in each original light field image of the checkerboard.

[0115] Traverse all the pixels of each original light field image of the checkerboard, filter out the qualified straight lines, and calculate the coordinates of the intersection of the straight lines, which are the corner coordinates of the original light field image of the checkerboard, as Figure 5 shown.

[0116] Since the parameters of the light field camera are different when shooting different original light field images of the checkerboard, and the corresponding microarray grid structures are also different, more accurate line features and corner coordinates can be obtained by adjusting the algorithm parameters. If the camera parameter settings are locked when shooting original light field images of the checkerboard at different angles, the same set of algorithm parameters can be directly used for calibration calculation, which can save calibration time.

[0117] S6: Establish the mapping relationship between the pixel coordinate system and the world coordinate system, and calibrate the parameters of the light field camera in combination with the corner coordinates of each original light field image of the checkerboard.

[0118] The mapping relationship between the pixel coordinate system and the world coordinate system is as Figure 6 shown.

[0119] The formula for calibrating the parameters of the light field camera is as follows:

[0120] ;

[0121]

[0122] where is the image coordinate system, is the camera coordinate system, is the world coordinate system, is the internal parameter of the light field camera, is the rotation matrix of the light field camera, is the translation vector of the light field camera, is the focal length of the light field camera in the horizontal direction, is the focal length of the light field camera in the vertical direction.

[0123] It should be understood that the various forms of the processes shown above can be reordered, added, or deleted. For example, the steps described in the disclosure of the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved, and no limitations are imposed herein.

[0124] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A light field camera calibration method based on line feature detection of the original light field image, characterized in that It includes the following steps: S1: Obtain the original light field image set of a white background taken by a light field camera at different focal lengths and different brightness levels, and process the original light field image set of the white background to obtain the projection center grid data of the microlens array; S2: Obtain the original light field image set of a checkerboard taken by a light field camera at different focal lengths, different angles, and different brightness levels; S3: Pair the original light field image set of the checkerboard with the original light field image set of the white background, and load the projection center grid data of the microlens array onto each original light field image of the checkerboard; S4: For each original light field image of the checkerboard loaded with the projection center grid data of the microlens array, calculate the gradient value of each pixel in the original light field image of the checkerboard, queue the gradient values of the pixels, select seed points, perform regional diffusion and model fitting in the direction of similar gradient angles, and obtain the line features in each original light field image of the checkerboard; In step S4, the process of queuing the gradient values of the pixels is as follows: First, find the maximum value among a large number of gradient values ; Secondly, based on the maximum value of the gradient value calculate the rank coefficient , and according to the rank coefficient perform gradient rank division on the pixels; among them, the calculation formula of the rank coefficient is as follows: where s is a preset level coefficient, ; Furthermore, the gradient levels are quantized, and the gradient value of each pixel after quantization is as follows: ； S5: Calculate the corner coordinates of each original light field image of the checkerboard according to the line features in each original light field image of the checkerboard; S6: Establish the mapping relationship between the pixel coordinate system and the world coordinate system, and calibrate the parameters of the light field camera in combination with the corner coordinates of each original light field image of the checkerboard.

2. The light field camera calibration method based on the line feature detection of the original light field image according to claim 1, characterized in that In step S1, generate the initial projection center grid coordinates of the microlens array according to the original light field image of the white background, update the projection center position of the microlens array using the interpolation calculation and weighted average method, and generate the projection center grid data of the microlens array.

3. The light field camera calibration method based on the detection of line features of the original light field image according to claim 1, wherein In step S4, the calculation formula for the gradient value of the pixel is as follows: Among them, is the two-dimensional image of the original light field image of the checkerboard grid, is the gradient value of the pixel, is the gradient direction of the pixel.

4. The light field camera calibration method based on the line feature detection of the original light field image according to claim 1, characterized in that After queuing the gradient values of the pixels, set the gradient threshold , suppress the gradient values less than the gradient threshold . The calculation formula of the gradient threshold is as follows: Among them, is the gradient direction angle tolerance value for regional growth, which is an empirical value.

5. The light field camera calibration method based on the line feature detection of the original light field image according to claim 1, characterized in that The process of selecting seed points and performing regional diffusion and model fitting in the direction of similar gradient angles to obtain the line features in each original light field image of the checkerboard is as follows: First, select the pixels that meet the threshold requirements and have eight neighborhoods from among many pixels as seed points; Secondly, for the gradient angle similarity direction of the seed points, the region growing algorithm is used to merge the seed points with approximately the same direction in the field to obtain the line support region ; Among them, is the pixel within the line support region, is the gradient direction angle of the i-th pixel within the line support region; Then, the line support region is abbreviated as r , and for the pixels r within the line support region j , the coordinates are , and the gradient value of the pixel j is . Based on the coordinates and gradient value of the pixel j , calculate the center point of the minimum bounding rectangle; among them, the calculation formula for the center point of the minimum bounding rectangle is as follows: Next, fit the pixels within the line support region r The fitting formula is as follows: Among them, is a symmetric positive semi-definite matrix; is the variance in the horizontal direction weighted by pixel gradients, is the variance in the vertical direction weighted by pixel gradients, is the covariance weighted by pixel gradients; Finally, use the evaluation parameters of the line features to estimate the error of the fitting result. If the error meets the conditions, record the fitting result as a straight line; Among them, the evaluation parameters include the angle tolerance, the maximum / minimum straight line length, the number of false alarms, and the line detection sensitivity.

6. The light field camera calibration method based on line feature detection of the original light field image according to claim 5, characterized in that, In step S5, traverse all the pixels of each original light field image of the checkerboard, filter out the qualified straight lines, and calculate the intersection coordinates of the straight lines, which are the corner coordinates of the original light field image of the checkerboard.

7. The light field camera calibration method based on line feature detection of the original light field image according to claim 1, wherein In step S6, the formula for calibrating the parameters of the light field camera is as follows: ； Among them, is the image coordinate system, is the camera coordinate system, is the world coordinate system, is the internal parameter of the light field camera, is the rotation matrix of the light field camera, is the translation vector of the light field camera, is the focal length of the light field camera in the horizontal direction, is the focal length of the light field camera in the vertical direction.

Images