Light field camera calibration method based on original light field image line feature detection

Abstract

The invention relates to the field of light field imaging, in particular to a light field camera calibration method based on original light field image line feature detection, which comprises the following steps: acquiring a white background original light field image set of a light field camera under different shooting conditions, and processing to obtain projection center grid data of a microlens array; obtaining a checkerboard original light field image set of the light field camera under different shooting conditions, and loading projection center grid data of the micro-lens array on each checkerboard original light field image; calculating a gradient value of each pixel in each checkerboard original light field image, queuing the gradient values of the pixels, and performing region diffusion and model fitting according to a gradient angle similar direction to obtain line features in each checkerboard original light field image so as to calculate angular point coordinates of each checkerboard original light field image; and constructing a light field camera projection model, and calibrating parameters of the light field camera in combination with the angular point coordinates. According to the method, introduced errors are reduced while time is saved.

Description

Technical Field

[0001] The present invention belongs to the technical field of light field imaging, and in particular relates to a light field camera calibration method based on original light field image line feature detection. Background Art

[0002] As an emerging imaging method, light field imaging technology can capture the three-dimensional spatial distribution information of light in a scene, providing a new research direction in the field of computer vision. However, due to the high-dimensional characteristics of the original light field image (including spatial information and angle information), the traditional calibration method based on two-dimensional images is difficult to directly apply to the calibration of light field cameras.

[0003] The traditional calibration method for light field cameras is to pre-process the original light field image by de-vignetting, rotation, translation, de-mosaicing, resampling, and interpolation to obtain a sub-aperture image, and then perform corner detection and calibration on the sub-aperture image. Since the process is cumbersome and the original light field image is not directly calibrated, it is time-consuming and prone to errors.

[0004] At present, there are some raw light field image line feature detection algorithms based on template matching. Such algorithms detect straight lines or features in raw light field images by matching with predefined templates. However, it is necessary to manually design predefined templates based on the structure of microarray light field cameras and light field cameras; for raw light field images of larger sizes or multi-template matching, there are problems of high overhead and low computational efficiency; there are also problems of sensitivity to noise and poor adaptation to scale and rotation changes. In particular, when the background in the raw light field image is complex or the difference between the target object and the background is not obvious, template matching may result in matching errors. Summary of the invention

[0005] In view of this, the present invention aims to provide a light field camera calibration method based on original light field image line feature detection, comprising the following steps: S1: obtaining a set of original light field images of a white background taken by a light field camera at different focal lengths and different brightnesses, and processing the set of original light field images of the white background to obtain projection center grid data of the microlens array; S2: Obtain a set of original light field images of a checkerboard taken by a light field camera at different focal lengths, different angles, and different brightnesses; S3: Pairing the chessboard original light field image set with the white background original light field image set, and loading the projection center grid data of the microlens array on each chessboard original light field image; S4: for each chessboard original light field image loaded with projection center grid data of the microlens array, calculate the gradient value of each pixel in the chessboard original light field image, arrange the gradient values ​​of the pixels, select seed points to perform regional diffusion and model fitting in the direction of similar gradient angles, and obtain line features in each chessboard original light field image; S5: Calculate the corner point coordinates of each chessboard original light field image according to the line features in each chessboard original light field image; S6: Establish a mapping relationship between the pixel coordinate system and the world coordinate system, and calibrate the parameters of the light field camera based on the coordinates of the corner points of each chessboard original light field image.

[0006] Furthermore, in step S1, the initial projection center grid coordinates of the microlens array are generated according to the white background original light field image, and the projection center position of the microlens array is updated using interpolation calculation and weighted average method to generate the projection center grid data of the microlens array.

[0007] Furthermore, in step S4, the calculation formula of the gradient value of the pixel is as follows:

[0008]

[0009] in, is the two-dimensional image of the original light field image of the chessboard, is the gradient value of the pixel, is the gradient direction of the pixel.

[0010] Furthermore, in step S4, the process of queuing the gradient values ​​of the pixels is as follows: First, find the maximum value of the gradient from many gradient values ; Secondly, based on the maximum value of the gradient Calculate the grade coefficient , according to the grade coefficient The pixels are divided into gradient levels; among them, the level coefficient The calculation formula is:

[0011] Among them, s is the preset level coefficient, ; Furthermore, the gradient level is quantized, and the gradient value of each pixel after quantization is for:

[0012] Furthermore, after queuing the gradient values ​​of the pixels, the gradient threshold is set , for values ​​less than the gradient threshold The gradient value is suppressed, the gradient threshold The calculation formula is:

[0013] in, is the gradient direction angle tolerance value of region growth, For experience value.

[0014] Furthermore, the seed points are selected to perform regional diffusion and model fitting in the direction of similar gradient angles, and the process of obtaining the line features in each chessboard original light field image is as follows: First, pixels that meet the threshold requirements and have eight neighborhoods are selected from a large number of pixels as seed points; Secondly, for the similar gradient angle direction of the seed points, the region growing algorithm is used to merge the seed points with approximately the same direction in the field to obtain the line support domain ;

[0015] in, are pixels within the line support domain, is the gradient direction angle of the i-th pixel in the line support domain; Then, the line support domain Abbreviated as r , line support domain r Internal Pixel j The coordinates of , pixels j The gradient value is , according to the pixel j The coordinates and gradient values ​​are used to calculate the center point of the minimum bounding rectangle. ; Among them, the center point of the minimum enclosing rectangle The calculation formula is as follows:

[0016] Next, the line support area r The pixels within are fitted, and the fitting formula is as follows:

[0017]

[0018]

[0019]

[0020] in, is a symmetric positive semidefinite matrix; is the horizontal variance of the weighted pixel gradient, is the vertical variance of the weighted pixel gradient, is the weighted covariance of pixel gradients; Finally, the error of the fitting result is estimated using the evaluation parameters of the line feature. If the error meets the conditions, the fitting result is recorded as a straight line. The evaluation parameters include angle tolerance, maximum / minimum straight line length, number of false alarms, and line detection sensitivity.

[0021] Furthermore, in step S5, all pixels of each chessboard original light field image are traversed, and the straight lines that meet the conditions are screened out, and the coordinates of the intersection of the straight lines are calculated, which are the coordinates of the corner points of the chessboard original light field image.

[0022] Furthermore, in step S6, the formula for calibrating the parameters of the light field camera is as follows: ;

[0023] in, is the image coordinate system, is the camera coordinate system, is the world coordinate system, is the internal parameter of the light field camera, is the rotation matrix of the light field camera, is the translation vector of the light field camera, is the horizontal focal length of the light field camera, is the vertical focal length of the light field camera.

[0024] Compared with the prior art, the invention can achieve the following beneficial effects: 1. The present invention can use the original light field image for calibration, avoiding the tedious process from the original light field image to the sub-aperture image, saving time and reducing the introduced errors.

[0025] 2. Detection is performed directly in the line feature area of ​​the original light field image, which provides richer image information and reduces the errors that may be introduced by resampling.

[0026] 3. Compared with the traditional template matching original light field image line feature detection algorithm, the present invention does not need to preset templates and perform matching, making the calibration process more efficient.

[0027] 4. For different light field camera settings and shooting conditions, the algorithm parameters of the model fitting can be adjusted to make it more adaptable to the environment.

[0028] 5. The model fitting algorithm can enhance the detection ability of areas with weak line features by adjusting the threshold, and the search ability is stronger.

[0029] 6. After searching the entire original light field image, determine the pixel with the maximum gradient, divide it into levels, and then use the region growing algorithm to set parameter thresholds such as tolerance to flexibly detect all reasonable straight lines in the original light field image without predefined templates.

[0030] 7. The original light field image line feature detection algorithm based on model fitting is relatively simple to implement and does not require too many parameter adjustments. At the same time, the algorithm has relatively few parameters, so it is easy to tune in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings: Figure 1 It is a flowchart of a light field camera calibration method based on original light field image line feature detection according to an embodiment of the invention.

[0032] Figure 2 It is a schematic diagram of projection center grid data of a microlens array according to an embodiment of the invention.

[0033] Figure 3 It is a schematic diagram of the process of detecting line features of original light field images based on model fitting according to an embodiment of the invention.

[0034] Figure 4 It is a schematic diagram of the detection results of the line feature area of ​​the chessboard original light field image described in the embodiment of the invention.

[0035] Figure 5 Schematic diagram of the coordinates of the corner points of the chessboard original light field image according to the embodiment of the invention.

[0036] Figure 6 Schematic diagram of a projection model of a light field camera according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0038] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0039] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0040] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0041] The present invention will be described in detail below with reference to the drawings and in combination with embodiments.

[0042] like Figure 1 As shown, the present invention provides a method for calibrating a light field camera based on original light field image line feature detection, comprising the following steps: S1: Obtain a set of original light field images of a white background taken by a light field camera at different focal lengths and different brightnesses, and process the set of original light field images of the white background to obtain projection center grid data of the microlens array.

[0043] Obtain pure white background light field files taken by light field cameras at different focal lengths and different brightness, decode the pure white background light field files, obtain the original white background light field image, and obtain the physical parameters of the original white background light field image, including the focal length, white balance and micro-array size of the light field camera at this time (the micro-array size includes the size of the micro-lens and the spacing between the micro-lenses). Generate the initial projection center grid coordinates of the micro-lens array according to the original white background light field image, and use interpolation calculation and weighted average method to update the projection center position of the micro-lens array, such as Figure 2 As shown, a grid coordinate file is generated, which is the projection center grid data of the microlens array in .mat format.

[0044] S2: Obtain a set of original light field images of a checkerboard taken by a light field camera at different focal lengths, different angles, and different brightness.

[0045] Since the focal length of the light field camera is variable, there are differences in the projection center grid of the microlens array under different focal length settings.

[0046] S3: Pair the checkerboard original light field image set with the white background original light field image set, and load the projection center grid data of the microlens array on each checkerboard original light field image.

[0047] The present invention uses the load function of matlab to read the grid coordinate file.

[0048] S4: For each checkerboard original light field image loaded with the projection center grid data of the microlens array, the gradient value of each pixel in the checkerboard original light field image is calculated, the gradient values ​​of the pixels are arranged in order, and seed points are selected for regional diffusion and model fitting in the direction of similar gradient angles to obtain line features in each checkerboard original light field image.

[0049] The specific process of step S4 is as follows Figure 3 shown.

[0050] The calculation formula of the gradient value of a pixel is as follows:

[0051]

[0052] in, is the two-dimensional image of the original light field image of the chessboard, is the gradient value of the pixel, is the gradient direction of the pixel.

[0053] The process of queuing the gradient values ​​of pixels is: First, find the maximum value of the gradient from many gradient values .

[0054] Secondly, based on the maximum value of the gradient Calculate the grade coefficient , according to the grade coefficient The pixels are divided into gradient levels; among them, the level coefficient The calculation formula is:

[0055] Among them, s is the preset level coefficient, .

[0056] The classification is equivalent to classifying into discrete data, which can reduce the complexity of data processing. Another advantage of the classification is that it reduces the impact of noise on line detection, that is, it reduces the false lines caused by small noise being misjudged as valid edges.

[0057] Furthermore, the gradient level is quantized, and the gradient value of each pixel after quantization is for:

[0058] The quantized gradient level processes pixels from high to low, giving priority to generating seed points in significant edge areas to reduce invalid calculations. Pixels with similar gradient values ​​are classified into the same level to ensure the consistency of gradient values ​​during region growth and avoid splitting continuous edges due to minor differences. Through normalized level values ​​(rather than absolute gradient values), the algorithm can automatically adapt to images of different contrasts without manually adjusting the gradient threshold.

[0059] Since the checkerboard original light field image lacks obvious gradient changes in the smooth area, the edge detection algorithm cannot identify the effective edge, and is easily affected by noise to form false tiny gradients. Therefore, after queuing the gradient values ​​of the pixels, a gradient threshold is set to suppress the gradient values ​​less than the gradient threshold. The calculation formula is:

[0060] in, is the gradient direction angle tolerance value of region growth, is the experience value, and Set the corresponding value according to the actual situation.

[0061] The process of selecting seed points to perform regional diffusion and model fitting in the direction of similar gradient angles to obtain the line features in each chessboard original light field image is as follows: First, pixels that meet the threshold requirements and have eight neighborhoods are selected from a large number of pixels as seed points.

[0062] Secondly, for the similar gradient angle direction of the seed points, the region growing algorithm is used to merge the seed points with approximately the same direction in the field to obtain the line support domain ;

[0063] in, are pixels within the line support domain, is the gradient direction angle of the i-th pixel in the line support domain.

[0064] Then, the line support domain Abbreviated as r, line support domain r Internal Pixel j The coordinates of , pixels j The gradient value is , according to the pixel j The coordinates and gradient values ​​are used to calculate the center point of the minimum bounding rectangle. ; Among them, the center point of the minimum enclosing rectangle The calculation formula is as follows:

[0065] Next, the line support area r The pixels within are fitted, and the fitting formula is as follows:

[0066]

[0067]

[0068]

[0069] in, is a symmetric semi-positive definite matrix whose eigenvectors represent the main directions of the gradient distribution in the line support domain (such as the direction of the line or edge); It is the horizontal variance of the weighted pixel gradient, reflecting the distribution intensity of the pixel in the horizontal direction; It is the vertical variance of the weighted pixel gradient, reflecting the distribution intensity of the pixel in the vertical direction; It is the weighted covariance of pixel gradients, reflecting the joint distribution characteristics of pixels in the horizontal and vertical directions.

[0070] Finally, the error of the fitting result is estimated using the evaluation parameters of the line feature. If the error meets the conditions, the fitting result is recorded as a straight line, such as Figure 4 As shown, if the error does not meet the conditions, the fitting result is ignored; wherein the evaluation parameters include angle tolerance, maximum / minimum straight line length, number of false alarms and line detection sensitivity.

[0071] Angle tolerance: It is used to define the maximum angle deviation allowed between the detected straight line and the true edge or reference direction. If the angle tolerance is set larger, line segments that deviate from the true angle can be accepted. If the angle tolerance is smaller, slightly inclined line segments that actually exist may be missed.

[0072] Maximum / minimum straight line length: Setting the minimum straight line length can filter out straight lines generated by noise, but it may miss real short edges. If it is too long, it may cause multiple line segments to merge incorrectly.

[0073] Number of false alarms: used to indirectly prove whether the number of false alarms is set reasonably, as an evaluation criterion.

[0074] Number of false alarms The calculation formula is as follows:

[0075] Among them, p represents the probability that each pixel is a straight line, P=TP / (TP+FP), TP is the true positive in the actual detection, FP is the false positive in the actual detection, k represents the number of pixels supporting the current straight line hypothesis, and S represents the total number of line segment candidates that may be generated in the algorithm.

[0076] If the NFA value of a line candidate is lower than the set threshold, it will be considered a valid line.

[0077] Line detection sensitivity: determines the algorithm's response threshold to the strength of edge gradients, usually related to the threshold of the edge detector (such as Canny's low / high threshold).

[0078] The setting of evaluation parameters should be combined with the main lens structure, microlens array structure and sensor structure of the light field camera. The threshold of the evaluation parameters is set according to the microlens projection in the line feature area of ​​the checkerboard original light field image to ensure that the features in the line feature microlens macro pixel area can be detected and participate in the fitting.

[0079] S5: Calculate the corner point coordinates of each chessboard original light field image according to the line features in each chessboard original light field image.

[0080] Traverse all pixels of each chessboard original light field image, filter out qualified straight lines, and calculate the coordinates of the intersection of the straight lines, which are the coordinates of the corner points of the chessboard original light field image, such as Figure 5 shown.

[0081] Since the parameters of the light field camera are different when shooting different chessboard original light fields, the corresponding micro-array grid structures are also different. More accurate line features and corner point coordinates can be obtained by adjusting the algorithm parameters. If the camera parameter settings are locked when shooting chessboard original light field images at different angles, the same set of algorithm parameters can be used to directly perform calibration calculations, which can save calibration time.

[0082] S6: Establish a mapping relationship between the pixel coordinate system and the world coordinate system, and calibrate the parameters of the light field camera based on the coordinates of the corner points of each chessboard original light field image.

[0083] The mapping relationship between the pixel coordinate system and the world coordinate system is as follows Figure 6 shown.

[0084] The formula for calibrating the parameters of a light field camera is as follows: ;

[0085] in, is the image coordinate system, is the camera coordinate system, is the world coordinate system, is the internal parameter of the light field camera, is the rotation matrix of the light field camera, is the translation vector of the light field camera, is the horizontal focal length of the light field camera, is the vertical focal length of the light field camera.

[0086] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0087] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A light field camera calibration method based on original light field image line feature detection, characterized in that: The following steps are involved: S1: obtaining a set of original light field images of a white background taken by a light field camera at different focal lengths and different brightnesses, and processing the set of original light field images of the white background to obtain projection center grid data of the microlens array; S2: Obtain a set of original light field images of a checkerboard taken by a light field camera at different focal lengths, different angles, and different brightnesses; S3: Pairing the chessboard original light field image set with the white background original light field image set, and loading the projection center grid data of the microlens array on each chessboard original light field image; S4: for each chessboard original light field image loaded with projection center grid data of the microlens array, calculate the gradient value of each pixel in the chessboard original light field image, arrange the gradient values ​​of the pixels, select seed points to perform regional diffusion and model fitting in the direction of similar gradient angles, and obtain line features in each chessboard original light field image; S5: Calculate the corner point coordinates of each chessboard original light field image according to the line features in each chessboard original light field image; S6: Establish a mapping relationship between the pixel coordinate system and the world coordinate system, and calibrate the parameters of the light field camera based on the coordinates of the corner points of each chessboard original light field image.

2. The light field camera calibration method based on original light field image line feature detection according to claim 1, characterized in that: In step S1, the initial projection center grid coordinates of the microlens array are generated according to the white background original light field image, and the projection center position of the microlens array is updated using interpolation calculation and weighted average method to generate the projection center grid data of the microlens array.

3. The light field camera calibration method based on original light field image line feature detection according to claim 1, characterized in that: In step S4, the calculation formula of the gradient value of the pixel is as follows: in, is the two-dimensional image of the original light field image of the chessboard, is the gradient value of the pixel, is the gradient direction of the pixel.

4. The light field camera calibration method based on original light field image line feature detection according to claim 1, characterized in that: In step S4, the process of queuing the gradient values ​​of pixels is: First, find the maximum value of the gradient from many gradient values ; Secondly, based on the maximum value of the gradient Calculate the grade coefficient , according to the grade coefficient The pixels are divided into gradient levels; among them, the level coefficient The calculation formula is: Among them, s is the preset level coefficient, ; Furthermore, the gradient level is quantized, and the gradient value of each pixel after quantization is for: 。 5. The light field camera calibration method based on original light field image line feature detection according to claim 1, characterized in that: After queuing the gradient values ​​of the pixels, set the gradient threshold , for values ​​less than the gradient threshold The gradient value is suppressed, the gradient threshold The calculation formula is: in, is the gradient direction angle tolerance value of region growth, For experience value.

6. The light field camera calibration method based on original light field image line feature detection according to claim 1, characterized in that: The process of selecting seed points to perform regional diffusion and model fitting in the direction of similar gradient angles to obtain the line features in each chessboard original light field image is as follows: First, pixels that meet the threshold requirements and have eight neighborhoods are selected from a large number of pixels as seed points; Secondly, for the similar gradient angle direction of the seed points, the region growing algorithm is used to merge the seed points with approximately the same direction in the field to obtain the line support domain ; in, are pixels within the line support domain, is the gradient direction angle of the i-th pixel in the line support domain; Then, the line support domain Abbreviated as r , line support domain r Internal Pixel j The coordinates of , pixels j The gradient value is , according to the pixel j The coordinates and gradient values ​​are used to calculate the center point of the minimum bounding rectangle. ; Among them, the center point of the minimum enclosing rectangle The calculation formula is as follows: Next, the line support area r The pixels within are fitted, and the fitting formula is as follows: in, is a symmetric positive semidefinite matrix; is the horizontal variance of the weighted pixel gradient, is the vertical variance of the weighted pixel gradient, is the weighted covariance of pixel gradients; Finally, the error of the fitting result is estimated using the evaluation parameters of the line feature. If the error meets the conditions, the fitting result is recorded as a straight line. The evaluation parameters include angle tolerance, maximum / minimum straight line length, number of false alarms, and line detection sensitivity.

7. The light field camera calibration method based on original light field image line feature detection according to claim 6, characterized in that: In step S5, all pixels of each chessboard original light field image are traversed, and the straight lines that meet the conditions are screened out, and the coordinates of the intersection of the straight lines are calculated, which are the coordinates of the corner points of the chessboard original light field image.

8. The light field camera calibration method based on original light field image line feature detection according to claim 1, characterized in that: In step S6, the formula for calibrating the parameters of the light field camera is as follows: ； in, is the image coordinate system, is the camera coordinate system, is the world coordinate system, is the internal parameter of the light field camera, is the rotation matrix of the light field camera, is the translation vector of the light field camera, is the horizontal focal length of the light field camera, is the vertical focal length of the light field camera.

Images