Ultra-wide-angle distortion calibration method and device

Abstract

The embodiment of the present invention discloses an ultra-wide-angle distortion calibration method and device, the calibration method includes calibration photography, calibration calculation and calibration comparison. The calibration device includes a test fixture, a calibration board and a calculation module. The technical solution provided by the embodiment of the present invention can perform fast and stable internal parameter calculation and full-field optical distortion correction, and restore the real world scene for the display device.

Description

[Technical field]

[0001] The present invention relates to the technical field of optical lens calibration, and in particular to an ultra-wide-angle distortion calibration method and device. [Background technology]

[0002] The field of view (FOV) of an ultra-wide-angle camera is generally 120 degrees or more, which is wider than that of an ordinary lens and can contain more information in one image. Ultra-wide angle enhances the sense of depth and space of the picture and is widely used in many fields such as video surveillance, robot navigation and 3D reconstruction. In the field of smart mobile terminals, ultra-wide angle is the basis of large aperture, multiple electronic zoom, smooth switching of multiple cameras, and image stitching, and is one of the main features of smartphone photography.

[0003] However, the imaging system of the ultra-wide-angle module introduces severe lens distortion while providing a wider range of imaging. This distortion causes the coordinates of the actual captured image to shift compared to the real image, for example, a captured straight line is distorted into an unknown curve, which limits the application of ultra-wide-angle.

[0004] According to the Brownian distortion model, lens distortion is divided into two main types: radial symmetric distortion and tangential distortion. Radial distortion includes barrel distortion and pincushion distortion; tangential distortion is sometimes also called eccentric distortion. Usually, ultra-wide-angle distortion is barrel distortion or a combination of multiple distortion forms, and the closer to the outer edge of the image, the greater the degree of distortion.

[0005] At present, the camera calibration scheme with a high level of automation generally requires the use of a calibration plate of a certain specification, identifying the two-dimensional (2D) points of the image through feature detection, and matching them one-to-one with the three-dimensional (3D) points of the space with known coordinates, finding the correspondence between 2D and 3D and completing the calibration. Considering that the images taken with ultra-wide angles have some unknown degree of deformation and distortion, solving the correspondence between 2D and 3D has become one of the pain points of ultra-wide angle camera calibration.

[0006] In the U.S. patent application number US7912317B2 and the Chinese patent application number CN109618090A, a method for distortion matching is provided, which finds the corresponding distortion parameters by adding a graphic object carrying known distortion to the image taken by the camera, and performs distortion correction on the entire image according to the distortion parameters obtained by table matching. First of all, the application scope of this technology is small. The distortion model must be existing, and it is difficult to adapt the distortion parameters and models that are not in the known distortion table; then, the distortion parameters in the known distortion table may not be very consistent with the real ultra-wide-angle camera lens, and this technology is not suitable for high-precision distortion correction. In addition, this technology can only correct the distortion of ultra-wide-angle images, but cannot obtain the camera intrinsic parameters of ultra-wide angles.

[0007] In a Chinese patent application with the patent number CN104077768A, a method for feature fitting of a distorted image is provided to complete camera calibration. For example, the fitted geometric shape is a curve segment, which corresponds to a straight line segment in the scene being photographed by the camera. This technology requires feature point fitting of the distorted image, such as fitting discrete points into curve segments. This process will involve a trade-off between calibration efficiency and accuracy. The feature points on the calibration plate used cannot be too dense, otherwise the solution efficiency will be low. Sparse feature points will lead to poor calibration result accuracy and large distortion coefficient error. [Summary of the invention]

[0008] In view of this, an embodiment of the present invention provides a method for solving the above-mentioned technical problems existing in the prior art.

[0009] In a first aspect, an embodiment of the present invention provides an ultra-wide-angle distortion calibration method, which includes the following steps: calibration photography, controlling a camera to photograph a calibration plate to obtain a calibration image a predetermined number of times; calibration calculation, using an ultra-wide-angle calibration algorithm to perform calculations on the obtained calibration image to obtain a calibration result; calibration comparison, if the calibration result meets a preset condition, outputting the calibration result; if the calibration result does not meet the preset condition, returning to re-execute the calibration photography step.

[0010] Through the solution provided in this embodiment, by utilizing the matching accuracy and simplified correction function of the ultra-wide-angle calibration algorithm, fast and high-precision ultra-wide-angle calibration can be performed. The calibration result includes not only internal parameters but also distortion parameters, and optical distortion correction of the entire field of view can be performed.

[0011] In a preferred embodiment, the calibration calculation step includes the following steps: constructing a homography matrix, taking five image feature points at the center of the field of view of the calibration image and obtaining two-dimensional coordinates, obtaining three-dimensional coordinates of the calibration feature points on the calibration plate corresponding to the five image feature points, establishing a two-dimensional feature point set and a three-dimensional feature point set, the image feature points in the two-dimensional feature point set and the three-dimensional feature point set correspond one-to-one to the calibration feature points, and constructing a homography matrix according to the two-dimensional coordinates and the three-dimensional coordinates; inverse sparse correction, inputting initial distortion parameters, extracting some of the image feature points within the image range of the calibration image A correction operation is performed on the two-dimensional image points to remove the ultra-wide-angle distortion of the two-dimensional image points to obtain two-dimensional ideal image points; corresponding to the feature point set, the two-dimensional ideal image points are homographically transformed by using the homography matrix to obtain three-dimensional ideal image points, and the three-dimensional ideal image points are feature-point matched with the basic feature points on the calibration plate to obtain the correspondence between the two-dimensional feature point set and the three-dimensional feature point set; calibration parameters are solved to obtain the matched and optimized two-dimensional feature point set and the three-dimensional feature point set according to the correspondence, and the homography matrix is ​​obtained accordingly, and the internal parameters and distortion parameters are optimized and solved.

[0012] Through the solution provided in this embodiment, the two-dimensional feature point set composed of the two-dimensional coordinates of the image feature points in the calibration image can be corresponded one-to-one with the three-dimensional feature point set composed of the three-dimensional coordinates of the corresponding calibration feature points on the calibration plate through the constructed homography matrix, thereby achieving high-precision calibration calculation, and the final internal parameters and distortion parameters can be solved through matching and optimization and output as the calibration result.

[0013] In a preferred embodiment, the feature point matching adopts a random sampling consensus algorithm or an iterative closest point algorithm.

[0014] Through the solution provided in this embodiment, the error of feature point matching is continuously corrected by multiple iterative operations of the random sampling consistency algorithm to improve the matching accuracy; the iterative nearest point algorithm is used to achieve accurate matching through continuous iterative minimization, thereby obtaining better matching effects efficiently and robustly.

[0015] In a preferred embodiment, the calibration and photographing step includes the following steps: assembling the camera onto a test fixture and starting the calibration software; rotating the calibration plate to a specified position and stabilizing it; calculating a photographing position corresponding to the specified position and moving the camera to the photographing position; controlling the camera to take pictures, and if the number of photographs exceeds the predetermined number, executing the calibration calculation step; if the number of photographs does not reach the predetermined number, re-executing the calibration and photographing step; wherein the number of photographs is at least three times.

[0016] Through the solution provided in this embodiment, the user can freely move the camera and rotate the calibration plate according to the actual camera model and user needs, and determine the position and angle of the camera through calculation according to the rotation angle and position of the calibration plate, so that the field of view of the camera can basically cover the entire calibration plate, laying a good foundation for higher-precision calibration.

[0017] In a preferred embodiment, the calibration comparison step includes the following steps: calculating the calibration error and comparing it with the preset conditions; judging the calibration result in combination with the calibration error, if the calibration result meets the preset conditions, outputting the calibration result, if the calibration result does not meet the preset conditions, returning to re-execute the calibration and photographing step and re-calibrating.

[0018] Through the scheme provided in this embodiment, the error calculation is performed on the calibration result obtained by the ultra-wide-angle calibration algorithm, and the calculated calibration error is compared with the preset conditions as the final judgment of the entire calibration method process, so that the output calibration result that meets the preset conditions has high precision and can successfully correct the distortion of the original image taken by the ultra-wide-angle camera.

[0019] In a second aspect, an embodiment of the present invention discloses an ultra-wide-angle distortion calibration device, which includes a test jig, a calibration plate and a computing module; the test jig and the calibration plate are respectively fixed and separated by a certain distance, the test jig can move relative to the calibration plate, the calibration plate has a fixed rotating shaft, and the calibration plate can rotate around the fixed rotating shaft; the test jig is used to fix a camera, the center of the camera is aligned with the center of the calibration plate, and the computing module is installed on the test jig and is communicatively connected to the camera.

[0020] Through the solution provided by this embodiment, the matching accuracy and simplified correction of the ultra-wide-angle calibration algorithm can be used to perform fast and high-precision ultra-wide-angle calibration. The calibration result includes not only internal parameters but also distortion parameters, and optical distortion correction of the entire field of view can be performed. During operation, the user can also adjust the position and rotation angle of the test fixture and calibration plate according to the camera model and actual needs, ultimately achieving the purpose of fast, efficient, and high-precision calibration.

[0021] In a preferred embodiment, the calibration plate is provided with a plurality of calibration feature points arranged in an array, and five image feature points are provided at the central area of ​​the calibration plate within the field of view of the camera.

[0022] Through the solution provided by this embodiment, calibration feature points and image feature points of different shapes are used as the basis for calibration, and the internal parameters and distortion parameters of barrel distortion can be calculated through the feature points in the field of view, thereby completing the correction of barrel distortion. In this preferred solution, compared with dense correction of all pixels of the entire image, the sparse correction operation based on image feature points can save computing resources and increase the calibration speed while ensuring distortion correction.

[0023] In a preferred embodiment, the density of the calibration points in the field of view is not less than 37×27.

[0024] Through the solution provided in this embodiment, sparse correction is only performed on the image feature points in the extracted calibration image, so as to facilitate high precision and high efficiency of ultra-wide-angle distortion calculation.

[0025] In a preferred embodiment, the array is arranged in the form of a checkerboard or a circular grid.

[0026] Through the solution provided in this embodiment, the calibration feature points and image feature points in the field of view can be clearly distinguished. For example, in a chessboard arrangement, the image feature points can be black grids or white grids that are different from the calibration feature points.

[0027] In a preferred embodiment, the calibration feature points are solid dots, and the image feature points are circular rings.

[0028] Through the solution provided in this embodiment, during the image capture and recognition process, the circular ring can be clearly distinguished from the solid circle due to its hollowness and larger diameter, thereby facilitating the recognition and calculation of image feature points after calibration and photography.

[0029] In a preferred embodiment, the calculation module includes a construction unit, a correction unit, a correspondence unit and a solution unit; the construction unit is used to construct a homography matrix, take five image feature points at the center of the field of view of the calibration image and obtain two-dimensional coordinates, obtain the three-dimensional coordinates of the calibration feature points on the calibration plate corresponding to the five image feature points, establish a two-dimensional feature point set and a three-dimensional feature point set, the image feature points in the two-dimensional feature point set and the three-dimensional feature point set correspond to the calibration feature points one by one, and construct a homography matrix according to the two-dimensional coordinates and the three-dimensional coordinates; the correction unit is used for inverse sparse correction, inputs the initial distortion parameters, extracts the calibration image, and calculates the corresponding three-dimensional coordinates of the calibration feature points. The method comprises the following steps: performing a correction operation on a plurality of two-dimensional image points within an image range of the image, removing the ultra-wide-angle distortion of the two-dimensional image points to obtain two-dimensional ideal image points; the corresponding unit is used for corresponding feature point sets, performing homography transformation on the two-dimensional ideal image points by using the homography matrix to obtain three-dimensional ideal image points, performing feature point matching on the three-dimensional ideal image points and the calibration feature points to obtain a correspondence between the two-dimensional feature point set and the three-dimensional feature point set; the solving unit is used for solving calibration parameters, obtaining the matched and optimized two-dimensional feature point set and the three-dimensional feature point set according to the correspondence, obtaining the homography matrix accordingly, and optimizing and solving the internal parameters and distortion parameters.

[0030] Through the solution provided in this embodiment, the two-dimensional feature point set composed of the two-dimensional coordinates of the image feature points in the calibration image can be corresponded one-to-one with the three-dimensional feature point set composed of the three-dimensional coordinates of the corresponding calibration feature points on the calibration plate through the constructed homography matrix, thereby achieving high-precision calibration calculation, and the final internal parameters and distortion parameters can be solved through matching and optimization and output as the calibration result.

[0031] Compared with the prior art, this technical solution has at least the following beneficial effects:

[0032] The ultra-wide-angle distortion calibration method and device disclosed in the embodiments of the present invention can perform fast and stable internal parameter calculation and full-field optical distortion correction, thereby restoring the real world scene for the display device.

Brief Description of the Drawings

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 Schematic diagram of the steps of the ultra-wide-angle distortion calibration method provided in Embodiment 1 of the present invention;

[0035] Figure 2 1 is a schematic diagram of the calibration calculation steps in the ultra-wide-angle distortion calibration method provided in Example 1 of the present invention;

[0036] Figure 3 1 is a schematic diagram of the steps of calibrating photography in the ultra-wide-angle distortion calibration method provided in Example 1 of the present invention;

[0037] Figure 4 is a schematic diagram of calibration comparison steps in the ultra-wide-angle distortion calibration method provided in Example 1 of the present invention;

[0038] Figure 5 is a specific flow chart of the ultra-wide-angle distortion calibration method provided in Embodiment 1 of the present invention;

[0039] Figure 6 is a schematic diagram of five image feature points on a calibration plate located within the field of view of a camera in the calibration calculation in the ultra-wide-angle distortion calibration method provided in Embodiment 1 of the present invention;

[0040] Figure 7 is an image schematic diagram of the inverse sparse correction in the calibration calculation in the ultra-wide-angle distortion calibration method provided in Embodiment 1 of the present invention;

[0041] Figure 8a is a distorted image before calibration using the ultra-wide-angle distortion calibration method provided by Embodiment 1 of the present invention;

[0042] Figure 8b is a corrected image after being calibrated using the ultra-wide-angle distortion calibration method provided by Embodiment 1 of the present invention;

[0043] Figure 8c is a distorted scene image before calibration using the ultra-wide-angle distortion calibration method provided by Embodiment 1 of the present invention;

[0044] Figure 8d is a corrected scene image after being calibrated using the ultra-wide-angle distortion calibration method provided by Embodiment 1 of the present invention;

[0045] Fig. 9 is a schematic structural diagram of an ultra-wide-angle distortion calibration device provided by Embodiment 2 of the present invention;

[0046] Fig.10 is a schematic diagram of a calibration plate in the ultra-wide-angle distortion calibration device provided in Example 2 of the present invention;

[0047] Fig.11 It is a schematic diagram of a calculation module in the ultra-wide-angle distortion calibration device provided in Example 2 of the present invention.

[0048] Reference numerals:

[0049] 1-Test tooling;

[0050] 2- Calibration plate;

[0051] 21-fixed shaft;

[0052] 22-calibrate feature points;

[0053] 23-image feature points;

[0054] 3- Computation module;

[0055] 31-tectonic unit;

[0056] 32-Correction unit;

[0057] 33-corresponding unit;

[0058] 34-Solving unit;

[0059] 4- Camera. [Specific implementation method]

[0060] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0061] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0063] like Figures 1 to 10 As shown, Figure 1 Schematic diagram of the steps of the ultra-wide-angle distortion calibration method provided in Embodiment 1 of the present invention; Figure 2 1 is a schematic diagram of the calibration calculation steps in the ultra-wide-angle distortion calibration method provided in Example 1 of the present invention; Figure 3 1 is a schematic diagram of the steps of calibrating photography in the ultra-wide-angle distortion calibration method provided in Example 1 of the present invention; Figure 4 is a schematic diagram of calibration comparison steps in the ultra-wide-angle distortion calibration method provided in Example 1 of the present invention; Figure 5 is a specific flow chart of the ultra-wide-angle distortion calibration method provided in Embodiment 1 of the present invention; Figure 6 is a schematic diagram of five image feature points on a calibration plate located within the field of view of a camera in the calibration calculation in the ultra-wide-angle distortion calibration method provided in Embodiment 1 of the present invention; Figure 7 is an image schematic diagram of the inverse sparse correction in the calibration calculation in the ultra-wide-angle distortion calibration method provided in Embodiment 1 of the present invention; Figure 8a is a distorted image before calibration using the ultra-wide-angle distortion calibration method provided by Embodiment 1 of the present invention; Figure 8b is a corrected image after being calibrated using the ultra-wide-angle distortion calibration method provided by Embodiment 1 of the present invention; Figure 8c is a distorted scene image before calibration using the ultra-wide-angle distortion calibration method provided by Embodiment 1 of the present invention; Figure 8d is a corrected scene image after being calibrated using the ultra-wide-angle distortion calibration method provided by Embodiment 1 of the present invention; Fig. 9 is a schematic structural diagram of an ultra-wide-angle distortion calibration device provided by Embodiment 2 of the present invention; Fig.10 is a schematic diagram of a calibration plate in the ultra-wide-angle distortion calibration device provided in Example 2 of the present invention; Fig.11 It is a schematic diagram of a calculation module in the ultra-wide-angle distortion calibration device provided in Example 2 of the present invention.

[0064] Example 1

[0065] like Figure 1 As shown, embodiment 1 of the present invention discloses an ultra-wide-angle distortion calibration method, which includes the following steps: calibration photography, controlling the camera 4 to photograph the calibration plate 2 to obtain a calibration image for a predetermined number of times; calibration calculation, using the ultra-wide-angle calibration algorithm to calculate the obtained calibration image to obtain a calibration result; calibration comparison, if the calibration result meets the preset conditions, then output the calibration result, if the calibration result does not meet the preset conditions, then return to re-execute the calibration photography step.

[0066] The ultra-wide-angle distortion calibration method of the present embodiment 1 utilizes the matching accuracy and simplified correction function of the ultra-wide-angle calibration algorithm to perform fast and high-precision ultra-wide-angle calibration. The calibration result includes not only internal parameters but also distortion parameters, and optical distortion correction of the entire field of view can be performed.

[0067] like Figure 2As shown, in the ultra-wide-angle distortion calibration method of the present embodiment 1, the calibration calculation step includes the following steps: constructing a homography matrix, taking five image feature points 23 at the center of the field of view of the calibration image and obtaining two-dimensional coordinates, obtaining three-dimensional coordinates of the calibration feature points 22 corresponding to the five image feature points 23 on the calibration plate 2, establishing a two-dimensional feature point set and a three-dimensional feature point set, the image feature points 23 in the two-dimensional feature point set and the three-dimensional feature point set correspond one to one with the calibration feature points 22, and constructing a homography matrix according to the two-dimensional coordinates and the three-dimensional coordinates; inverse sparse correction, inputting initial distortion parameters, extracting the calibration feature points 22, and performing a three-dimensional feature correction on the calibration plate 2. A correction operation is performed on several two-dimensional image points within the image range of a fixed image to remove the ultra-wide-angle distortion of the two-dimensional image points to obtain two-dimensional ideal image points; corresponding to the feature point set, the two-dimensional ideal image points are transformed homographically using the homography matrix to obtain three-dimensional ideal image points, and the three-dimensional ideal image points are matched with the basic feature points on the calibration plate to obtain the correspondence between the two-dimensional feature point set and the three-dimensional feature point set; calibration parameters are solved to obtain the matched and optimized two-dimensional feature point set and three-dimensional feature point set according to the correspondence, and the homography matrix is ​​obtained accordingly, and the intrinsic parameters and distortion parameters are optimized and solved.

[0068] In the ultra-wide-angle distortion calibration method of this embodiment 1, the two-dimensional feature point set composed of the two-dimensional coordinates of the image feature points 23 in the calibration image can be one-to-one corresponded with the three-dimensional feature point set composed of the three-dimensional coordinates of the corresponding calibration feature points 22 on the calibration plate 2 through the constructed homography matrix, thereby achieving high-precision calibration calculation, and the final internal parameters and distortion parameters can be solved through matching and optimization and output as calibration results.

[0069] In the ultra-wide-angle distortion calibration method of the first embodiment, feature point matching adopts a random sampling consensus algorithm or an iterative closest point algorithm.

[0070] The ultra-wide-angle distortion calibration method of this embodiment 1 uses multiple iterative operations of the random sampling consistency algorithm to continuously correct the error of feature point matching and improve the matching accuracy; it uses the iterative nearest point algorithm to achieve precise matching through continuous iterative minimization, thereby efficiently and robustly obtaining better matching effects.

[0071] like Figure 3 As shown, in the ultra-wide-angle distortion calibration method of this embodiment 1, the calibration photographing step includes the following steps: assembling the camera 4 onto the test fixture 1 and starting the calibration software; rotating the calibration plate 2 to a specified position and stabilizing it; calculating the photographing position corresponding to the specified position, and moving the camera 4 to the photographing position; controlling the camera 4 to take photos, and if the number of photographs exceeds a predetermined number, executing the calibration calculation step, and if the number of photographs does not reach the predetermined number, re-executing the calibration photographing step; wherein the number of photographs is at least three times.

[0072] In the ultra-wide-angle distortion calibration method of this embodiment 1, the user can freely move the camera 4 and rotate the calibration plate 2 according to the actual model of the camera 4 and the user's needs, and determine the position and angle of the camera 4 by calculation according to the rotation angle and position of the calibration plate 2, so that the field of view of the camera 4 can basically cover the entire calibration plate 2, laying a good foundation for higher-precision calibration.

[0073] like Figure 4 As shown, in the ultra-wide-angle distortion calibration method of the present embodiment 1, the calibration comparison step includes the following steps: calculating the calibration error and comparing it with the preset condition; judging the calibration result in combination with the calibration error, if the calibration result meets the preset condition, outputting the calibration result, if the calibration result does not meet the preset condition, returning to re-execute the calibration photography step and re-calibrating.

[0074] In the ultra-wide-angle distortion calibration method of the present embodiment 1, an error calculation is performed on the calibration result obtained by the ultra-wide-angle calibration algorithm, and the calculated calibration error is compared with the preset conditions as the final judgment of the entire calibration method process, so that the output calibration result that meets the preset conditions has high precision and can successfully correct the distortion of the original image taken by the ultra-wide-angle camera 4.

[0075] like Figure 5 As shown, the ultra-wide-angle distortion calibration method of this embodiment 1 takes the center of the circle as the image feature point and the center of the solid circle as the calibration feature point as an example. The calibration plate 2 is provided with a plurality of calibration feature points 22 arranged in an array, and five image feature points 23 are provided in the central area of ​​the calibration plate 2 within the field of view of the camera 4. The specific process is as follows:

[0076] 1. Assemble the ultra-wide-angle camera 4 onto the test fixture 1 and start the software to begin calibration.

[0077] 2. Control the calibration plate 2 to rotate to the specified position k and keep the calibration plate 2 stable; call the algorithm to calculate the code value c corresponding to the distance d k The motor for driving the ultra-wide-angle camera 4 moves the ultra-wide-angle camera 4 to c k Position, get the calibration image I k , passed to the calibration algorithm library;

[0078] 3. Determine the number of photos taken (at least 3 times). If the conditions are met, proceed to step 4, otherwise recalibrate;

[0079] 4. Execute the ultra-wide-angle calibration algorithm;

[0080] 5. Calculate the calibration error and compare it with the preset threshold value (i.e. the preset condition mentioned above). The calibration result is judged by the program. If the condition is met, go to step 6, otherwise recalibrate;

[0081] 6. Store the calibration results.

[0082] Among them, the detailed process of the ultra-wide-angle calibration algorithm is as follows:

[0083] 1. Constructing the homography matrix

[0084] In view of the fact that the degree of distortion of the image captured by the ultra-wide-angle camera 4 increases from the center of the field of view to the edge, five image feature points 23 are taken from the calibration plate 2 at the center of the field of view, such as Figure 6 The two-dimensional image coordinates of the five image feature points 23 are detected, combined with the three-dimensional coordinates of the corresponding calibration feature points 22 on the calibration plate 2, to construct the homography matrix H and solve it.

[0085] p mark =HP mark

[0086] Among them, the homography matrix H has 9 degrees of freedom. mark and P mark They are the set of two-dimensional image feature points and three-dimensional calibration feature points, each containing 5 points, and the points in the two point sets correspond one to one.

[0087] 2. Inverse Sparse Correction

[0088] Generally, the distortion parameters are the output results after camera calibration. In this embodiment 1, a set of initial distortion parameters is used as input, and only all feature points p identified within the entire image range are all Perform inverse sparse correction to obtain the two-dimensional ideal image point p′ without ultra-wide-angle distortion. all . Unlike other calibration methods that correct all pixels of the entire ultra-wide-angle image, the calibration method of this embodiment 1 only corrects the extracted sparse feature points, which can greatly improve efficiency and increase calibration capacity. It is worth noting that the input initial distortion parameters do not need to perfectly remove the distortion, but only to weaken the distortion effect caused by the ultra-wide-angle distortion. Therefore, it is sufficient to roughly adjust the feature points of the distorted distribution within the field of view of the ultra-wide-angle camera 4 to a real distribution state that is close to horizontal and vertical. The schematic diagram of the reverse sparse correction is shown in Figure 7 shown.

[0089] 3. Find the correspondence between two sets of 2D and 3D points

[0090] Combining the first and second steps, the two-dimensional ideal image point p′ without ultra-wide-angle distortion is all Perform homography transformation to obtain the three-dimensional ideal image point P′ all , and combined with the known three-dimensional calibration feature points P on the calibration plate 2 allPerform feature point matching. Some common feature point matching algorithms include the Random Sample Consensus Algorithm (RANSAC), the Iterative Closest Points Algorithm (ICP), etc. The calibration method of this embodiment 1 takes the random sampling consensus algorithm as an example, randomly selects 4 pairs of internal points to obtain the internal homography matrix H in By H in Test all remaining points and divide them into inside and outside points. Continue to randomly select 4 pairs of inside points to calculate H in , and continue this process until the number of iterations or iteration accuracy is met. At this time, all outliers are removed to complete the precise matching. And for the removed erroneous points, compensation and optimization are performed for the erroneous removed points. Combine the surrounding 15 adjacent points to calculate the local homography matrix H local , project the error point onto the calibration board and perform adaptive distance matching again. If it matches the 3D calibration feature point on the calibration board, the error-removed point is extracted.

[0091] 4. Calibration parameter solution

[0092] According to the matching relationship between the two-dimensional feature points and the three-dimensional feature points obtained in the third step, the matched and optimized two-dimensional feature point set p is obtained. all and the three-dimensional feature point set P all The homography matrix H is obtained according to the one-to-one correspondence between the two sets of point sets, and the maximum likelihood estimation and linear least squares method are used to optimize and solve the internal parameters and distortion parameters.

[0093] like Figure 8a As shown in 8b, in the image captured by the ultra-wide-angle camera 4 before calibration, the feature points originally arranged horizontally and vertically on the calibration plate 2 are severely distorted (especially obvious at the four corners). After calibration, as shown in 8b, the optical distortion of the full field of view of the ultra-wide-angle camera 4 is corrected, and the feature points are restored to the true horizontal and vertical arrangement. Figure 8c and Figure 8d This is a diagram of an application scenario before and after the ultra-wide-angle camera 4 is calibrated. Figure 8c and Figure 8d It can be seen that after distortion correction, the door frame is restored from a curve to a real straight line, and the correction effect is obvious.

[0094] Example 2

[0095] like Fig. 9As shown, embodiment 2 of the present invention discloses an ultra-wide-angle distortion calibration device, which includes a test fixture 1, a calibration plate 2 and a computing module 3; the test fixture 1 and the calibration plate 2 are respectively fixed and separated by a certain distance, the test fixture 1 can move relative to the calibration plate 2, the calibration plate 2 has a fixed rotating shaft 21, and the calibration plate 2 can rotate around the fixed rotating shaft 21; the test fixture 1 is used to fix the camera 4, the center of the camera 4 is aligned with the center of the calibration plate 2, and the computing module 3 is installed on the test fixture 1 and is communicatively connected with the camera 4.

[0096] In the ultra-wide-angle distortion calibration device of this embodiment 2, the ultra-wide-angle camera 4 is fixed by the test fixture 1, and the optical axis direction of the camera 4 is perpendicular to the calibration plate 2 in the initial position and intersects at the center point of the calibration plate 2. The distance between the ultra-wide-angle camera 4 and the center of the calibration plate 2 is d. The calibration plate 2 can rotate around the fixed rotation axis 21, and the width w and height h of the calibration plate 2 are not limited.

[0097] By utilizing the matching accuracy and simplified correction of the ultra-wide-angle calibration algorithm, fast and high-precision ultra-wide-angle calibration can be performed. The calibration result includes not only internal parameters but also distortion parameters, and optical distortion correction of the entire field of view can be performed. During operation, the user can also adjust the position and rotation angle of the test fixture 1 and the calibration plate 2 according to the model of the camera 4 and actual needs, ultimately achieving the purpose of fast, efficient, and high-precision calibration.

[0098] like Fig.10 As shown, in the ultra-wide-angle distortion calibration device of this embodiment 2, a plurality of calibration feature points 22 arranged in an array are provided on the calibration plate 2, and five image feature points 23 are provided in the central area of ​​the calibration plate 2 within the field of view of the camera 4, and the pattern of the feature points on the calibration plate is not limited.

[0099] In the ultra-wide-angle distortion calibration device of the second embodiment, calibration feature points 22 and image feature points 23 of different shapes are used as the basis for calibration, and the internal parameters and distortion parameters of the barrel distortion can be calculated through the feature points in the field of view, thereby completing the correction of the barrel distortion. In this preferred solution, compared with the dense correction of all pixels of the entire image, the sparse correction operation based on the image feature points 23 can save computing resources and improve the calibration speed while ensuring the distortion correction.

[0100] In the ultra-wide-angle distortion calibration device of the second embodiment, the density of the calibration points in the field of view is not less than 37×27.

[0101] In the ultra-wide-angle distortion calibration device of the present embodiment 2, there is a large difference between the number of calibration feature points 22 in the field of view and all pixels of the entire image, and only sparse correction is performed on the image feature points 23 in the extracted calibration image, which is beneficial for the camera 4 to extract the image feature points 23 in the calibration image, so as to facilitate the high precision and high efficiency of the ultra-wide-angle distortion operation.

[0102] In the ultra-wide-angle distortion calibration device of the second embodiment, the array is arranged in the form of a checkerboard or a circular grid.

[0103] In the ultra-wide-angle distortion calibration device of the second embodiment, the calibration feature points 22 and the image feature points 23 in the field of view can be clearly distinguished. For example, in a checkerboard arrangement, the image feature points 23 can be black or white grids that are different from the calibration feature points 22.

[0104] In the ultra-wide-angle distortion calibration device of the second embodiment, the calibration feature points 22 are solid dots, and the image feature points 23 are circular rings.

[0105] In the ultra-wide-angle distortion calibration device of the present embodiment 2, during the image capture and recognition process, the circular ring can be clearly distinguished from the solid circle due to its hollowness and larger diameter, thereby facilitating the recognition and calculation of the image feature points 23 after calibration photography.

[0106] like Fig.11 As shown, in the ultra-wide-angle distortion calibration device of this embodiment 2, the calculation module 3 includes a construction unit 31, a correction unit 32, a correspondence unit 33 and a solution unit 34; the construction unit 31 is used to construct a homography matrix, take five image feature points 23 at the center of the field of view of the calibration image and obtain two-dimensional coordinates, obtain the three-dimensional coordinates of the calibration feature points 22 corresponding to the five image feature points 23 on the calibration plate 2, establish a two-dimensional feature point set and a three-dimensional feature point set, the image feature points 23 in the two-dimensional feature point set and the three-dimensional feature point set correspond to the calibration feature points 22 one by one, and construct a homography matrix according to the two-dimensional coordinates and the three-dimensional coordinates; the correction unit 32 is used for inverse sparse correction, and the input Input initial distortion parameters, perform correction operations on several two-dimensional image points within the image range of the extracted calibration image, remove the ultra-wide-angle distortion of the two-dimensional image points to obtain two-dimensional ideal image points; the corresponding unit 33 is used for corresponding feature point sets, and uses the homography matrix to perform homography transformation on the two-dimensional ideal image points to obtain three-dimensional ideal image points, and perform feature point matching on the three-dimensional ideal image points and the calibration feature points 22 to obtain the correspondence between the two-dimensional feature point set and the three-dimensional feature point set; the solving unit 34 is used for solving calibration parameters, and obtains the matched and optimized two-dimensional feature point set and three-dimensional feature point set according to the correspondence, thereby obtaining the homography matrix, and optimizing and solving the internal parameters and distortion parameters.

[0107] In the ultra-wide-angle distortion calibration device of the present embodiment 2, the two-dimensional feature point set composed of the two-dimensional coordinates of the image feature points 23 in the calibration image can be corresponded one-to-one with the three-dimensional feature point set composed of the three-dimensional coordinates of the corresponding calibration feature points 22 on the calibration plate 2 through the constructed homography matrix, thereby achieving high-precision calibration calculation, and the final internal parameters and distortion parameters can be solved through matching and optimization and output as calibration results.

[0108] The ultra-wide-angle distortion calibration method and device disclosed in the embodiments of the present invention can perform fast and stable internal parameter calculation and full-field optical distortion correction, thereby restoring the real world scene for the display device.

[0109] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for ultra-wide-angle distortion calibration, characterized in that: The ultra-wide-angle distortion calibration method comprises the following steps: Calibration photography: controlling the camera to take photos of the calibration plate to obtain calibration images for a predetermined number of times; Calibration calculation, using an ultra-wide-angle calibration algorithm to calculate the acquired calibration image to obtain a calibration result; Calibration comparison, if the calibration result meets the preset conditions, then output the calibration result, if the calibration result does not meet the preset conditions, then return to re-execute the calibration and photography step; The calibration calculation step comprises the following steps: Constructing a homography matrix, taking five image feature points at the center of the field of view of the calibration image and obtaining two-dimensional coordinates, obtaining three-dimensional coordinates of calibration feature points on the calibration plate corresponding to the five image feature points, establishing a two-dimensional feature point set and a three-dimensional feature point set, wherein the image feature points in the two-dimensional feature point set and the three-dimensional feature point set correspond one-to-one to the calibration feature points, and constructing a homography matrix according to the two-dimensional coordinates and the three-dimensional coordinates; Inverse sparse correction, inputting initial distortion parameters, performing correction operations on a number of two-dimensional image points within the image range of the calibration image, removing ultra-wide-angle distortion of the two-dimensional image points to obtain two-dimensional ideal image points; Corresponding to the feature point set, using the homography matrix to perform homography transformation on the two-dimensional ideal image points to obtain three-dimensional ideal image points, performing feature point matching on the three-dimensional ideal image points and the basic feature points on the calibration plate to obtain a correspondence between the two-dimensional feature point set and the three-dimensional feature point set; The calibration parameters are solved, and the matched and optimized two-dimensional feature point set and three-dimensional feature point set are obtained according to the corresponding relationship, and the homography matrix is ​​obtained accordingly, and the internal parameters and distortion parameters are optimized and solved.

2. The ultra-wide-angle distortion calibration method according to claim 1, characterized in that: The feature point matching adopts a random sampling consensus algorithm or an iterative closest point algorithm.

3. The ultra-wide-angle distortion calibration method according to claim 1, characterized in that: The calibration and photographing step comprises the following steps: Assemble the camera onto the test fixture and start the calibration software; Rotate the calibration plate to the specified position and stabilize it; Calculating a photographing position corresponding to the designated position, and moving the camera to the photographing position; Controlling the camera to take photos, if the number of photos exceeds the predetermined number, executing the calibration calculation step, if the number of photos does not reach the predetermined number, re-executing the calibration photo taking step; Wherein, the number of photographs is at least three.

4. The ultra-wide-angle distortion calibration method according to claim 3, characterized in that: The calibration comparison step comprises the following steps: Calculating the calibration error and comparing it with the preset condition; The calibration result is interpreted in combination with the calibration error. If the calibration result meets the preset condition, the calibration result is output. If the calibration result does not meet the preset condition, the calibration and photographing step is returned to be re-executed to re-calibrate.

5. An ultra-wide angle distortion calibration device, characterized in that: The ultra-wide-angle distortion calibration device includes a test fixture, a calibration board and a calculation module; The test fixture and the calibration plate are respectively fixed and separated by a certain distance, the test fixture can move relative to the calibration plate, the calibration plate has a fixed rotating shaft, and the calibration plate can rotate around the fixed rotating shaft; The test fixture is used to fix the camera, the center of the camera is aligned with the center of the calibration plate, and the computing module is installed on the test fixture and is communicatively connected with the camera; The calculation module is used to perform calibration photography, control the camera to take photos of the calibration plate to obtain calibration images for a predetermined number of times; calibration calculation, use the ultra-wide-angle calibration algorithm to calculate the obtained calibration images to obtain calibration results; Calibration comparison: if the calibration result meets the preset conditions, the calibration result is output; if the calibration result does not meet the preset conditions, the calibration and photography steps are returned to be re-executed; The calculation module includes a construction unit, a correction unit, a corresponding unit and a solution unit; The construction unit is used to construct a homography matrix, take five image feature points at the center of the field of view of the calibration image and obtain two-dimensional coordinates, obtain three-dimensional coordinates of calibration feature points on the calibration plate corresponding to the five image feature points, establish a two-dimensional feature point set and a three-dimensional feature point set, the image feature points in the two-dimensional feature point set and the three-dimensional feature point set correspond one-to-one to the calibration feature points, and construct a homography matrix according to the two-dimensional coordinates and the three-dimensional coordinates; The correction unit is used for inverse sparse correction, inputting initial distortion parameters, performing correction operations on a number of two-dimensional image points within the image range of the calibration image, and removing the ultra-wide-angle distortion of the two-dimensional image points to obtain two-dimensional ideal image points; The corresponding unit is used for corresponding feature point sets, performing homography transformation on the two-dimensional ideal image points using the homography matrix to obtain three-dimensional ideal image points, performing feature point matching on the three-dimensional ideal image points and the calibrated feature points to obtain a corresponding relationship between the two-dimensional feature point set and the three-dimensional feature point set; The solving unit is used for solving calibration parameters, obtaining the matched and optimized two-dimensional feature point set and three-dimensional feature point set according to the corresponding relationship, obtaining the homography matrix based on the obtained data, and optimizing and solving the internal parameters and distortion parameters.

6. The ultra-wide-angle distortion calibration device according to claim 5, characterized in that: The calibration plate is provided with a plurality of calibration feature points arranged in an array, and five image feature points are provided at the central area of ​​the calibration plate within the field of view of the camera.

7. The ultra-wide-angle distortion calibration device according to claim 6, characterized in that: The density of the calibration points in the field of view is not less than 37×27.

8. The ultra-wide-angle distortion calibration device according to claim 6, characterized in that: The array is arranged in a form of a checkerboard or a circular grid.

9. The ultra-wide-angle distortion calibration device according to claim 6, characterized in that: The calibration feature points are solid dots, and the image feature points are circular rings.

Images