Camera extrinsic calibration result correction method and system based on OpenCV Eigen

Abstract

The present invention belongs to the field of camera calibration technology, and specifically relates to a method and system for correcting camera extrinsic calibration results based on OpenCV Eigen, including: obtaining original camera extrinsic parameters and image data captured by each camera to be corrected; constructing a pixel point coordinate determinant according to the image data; performing matrix multiplication on the pixel point coordinate determinant and the camera extrinsic parameter to obtain the camera mapping coordinate determinant; converting the image data into a grayscale image, extracting contour corner points in the overlapping area; extracting the brightness value of the contour corner point, and calculating the ratio sum of the brightness values ​​of each image; performing random perturbations on the position and posture parameters of the original camera extrinsic parameters to generate new camera extrinsic parameters; using the new camera extrinsic parameters to recalculate the ratio sum of the brightness values; and selecting the camera extrinsic parameter corresponding to the lower brightness value ratio sum as the correction result. The present invention adjusts the extrinsic parameter parameters and verifies the accuracy through a computer for multiple times to obtain more accurate camera extrinsic parameters.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera calibration, and in particular relates to a method and system for correcting camera extrinsic calibration results based on OpenCV Eigen. Background Art

[0002] When calibrating the camera's external parameters, there will be certain errors in the camera's external parameter calibration results due to the influence of small errors in camera resolution, clarity, capture algorithm, floating point value, etc. Figure 1 As shown in FIG. 1 , when multiple camera images are stitched together using external parameters, there may be certain deviations due to errors in calibrating the external parameters.

[0003] When there are obvious errors in the calibration results, we usually choose to adjust the algorithm and recalibrate. Since accurate camera external parameter calibration has high requirements for the calibration field and calibration board, recalibration may also result in worse calibration results, or multiple calibrations may be required in the calibration field. This results in low efficiency of camera external parameter calibration and is greatly restricted by the environment. Summary of the invention

[0004] Purpose of the invention: Since the calibrated external parameters required for stitching multiple camera images have high accuracy, the present invention provides a camera extrinsic parameter calibration result correction method and system based on OpenCV Eigen, which is based on OpenCV Eigen batch processing pixel point determinant mapping, and is independent of the calibration field while ensuring the CPU's image pixel processing performance. The extrinsic parameter parameters are adjusted and the accuracy is verified by the computer multiple times to obtain more accurate camera extrinsic parameters.

[0005] Technical solution: The method for correcting camera extrinsic calibration results based on OpenCV Eigen of the present invention comprises the following steps:

[0006] S1, obtaining camera extrinsic parameters and a stitching area camera image formed by calibrating images taken by multiple cameras with the camera extrinsic parameters, and generating a pixel point coordinate determinant according to the resolution of the stitching area camera image, wherein the pixel point coordinate determinant is used to represent the row and column position of each pixel point in the stitching area camera image in the image coordinate system;

[0007] S2. Convert the pixel points in the pixel point coordinate determinant from the image coordinate system to the camera coordinate system through the camera external parameter to obtain a mapping coordinate determinant;

[0008] S3, obtaining different camera mapping images under camera extrinsic configurations through the mapping coordinate determinant;

[0009] S4, converting the camera mapping image into a grayscale image by grayscale processing, and extracting contour corner points in the overlapping area of ​​the grayscale mapping images of different cameras by calculating the brightness gradient change to generate a feature point set of the overlapping area;

[0010] S5, calculating the brightness ratio difference of the feature point set in the overlapping area in different camera images, and accumulating the total brightness ratio difference as an evaluation index;

[0011] S6. Randomly perturb the position and attitude parameters of the camera extrinsic parameters to generate new camera extrinsic parameters, repeat steps S2 to S5, and use the new camera extrinsic parameters to recalculate the sum of brightness ratio differences;

[0012] S7. Select the camera external parameter corresponding to the lower sum of the brightness value ratios as the correction result.

[0013] To further improve the above technical solution, the step of generating the pixel coordinate determinant according to the resolution of the camera image in the stitching area in S1 includes:

[0014] Initialize each element in the pixel determinant matrix to 1;

[0015] Loop through the xy coordinates of each pixel in the camera image of the stitching area, and construct the pixel coordinate determinant by setting the first and second rows of the pixel determinant matrix to be the x coordinate and y coordinate of the pixel respectively, expressed as

[0016] .

[0017] Furthermore, S2 includes: decomposing the original camera extrinsic parameter matrix into a rotation matrix and a displacement vector, wherein the rotation matrix is ​​used to describe the rotation relationship between the image coordinate system and the camera coordinate system, and the displacement vector represents the position of the camera relative to the global coordinate system; multiplying the pixel point coordinate determinant with the rotation matrix and adding the displacement vector to obtain the mapping coordinate determinant of the pixel point of the camera image corresponding to the display screen.

[0018] Further, the S3 includes: using the mapping coordinate determinant, utilizing the remap function in the OpenCV library, and performing geometric correction on the converted image coordinates by a lookup table to achieve spatial calibration of the image, wherein the remap function interpolates and maps the pixel points in the image based on the mapping coordinate table of the camera extrinsic parameters, thereby generating a camera mapping image under the original camera extrinsic parameter matrix configuration.

[0019] Furthermore, the S4 includes: using the cvtColor function of the OpenCV library to convert each camera mapping image into a grayscale image; defining a container for storing contour corner points; determining the overlapping area in the grayscale image, traversing each pixel point in the overlapping area, checking whether the pixel point coordinates are valid, and if valid, calculating the grayscale difference of the pixel point in the horizontal and vertical directions and the norm of the gradient vector, and storing the contour corner points whose gradient vector norms meet the conditions in the container.

[0020] Furthermore, S5 includes: traversing the identified contour corner points, and performing calculations by accessing the brightness values ​​of corresponding pixels in the grayscale image; obtaining the brightness value of each contour corner point in two different camera images, and calculating the ratio of the brightness values; and accumulating the square sum of all ratios to obtain the sum of the ratios of the brightness values.

[0021] Furthermore, the S6 includes: generating random offsets for adjusting six parameters of the camera extrinsic parameters, including: roll angle, pitch angle, yaw angle and three spatial position coordinates; generating an incremental matrix for parameter adjustment; multiplying the incremental matrix with the original camera extrinsic parameters to obtain new camera extrinsic parameters.

[0022] Furthermore, the parameter adjustment of the camera extrinsic parameters and the calculation of the sum of the brightness ratio differences are iterated multiple times; the sum of the brightness ratio differences after each iteration is recorded; and the stitching effect of the overlapping area is optimized by minimizing the sum of the brightness ratio differences until the optimal camera extrinsic parameters are obtained.

[0023] Furthermore, the amplitude of the camera extrinsic parameter adjustment is generated by a random distribution function, and the adjustment amplitude decreases as the number of iterations increases.

[0024] The camera extrinsic parameter correction system for implementing the above method includes: a camera extrinsic parameter acquisition module, used to acquire the camera's extrinsic parameters and image data; a pixel point determinant generation module, used to generate a pixel point determinant according to the resolution of the camera image; an extrinsic parameter mapping module, used to perform matrix multiplication of the pixel point determinant with the camera extrinsic parameter to generate a camera mapping coordinate determinant; a contour corner point extraction module, used to convert the camera image into a grayscale image and identify the contour corner points in the overlapping area; an extrinsic parameter adjustment module, used to generate new camera extrinsic parameters by offsetting six parameters of the camera extrinsic parameter; an evaluation calculation module, used to calculate the brightness value of the contour corner point and generate a ratio sum; and a result output module, used to output the corrected extrinsic parameter matrix.

[0025] Beneficial effect: Compared with the prior art, the advantages of the present invention are as follows: After the extrinsic parameters of the vehicle-mounted camera are calibrated, the pixel coordinate determinant is multiplied by the calibrated extrinsic parameters using OpenCV Eigen to obtain the mapping coordinate determinant of the camera image and the display window. The camera image is converted into a grayscale image, and the overlapping area with other camera images is confirmed. In the area, a group of pixels with obvious changes, i.e., contour corner points, are selected by the brightness gradient change between the pixels. The sum of the brightness ratios of the camera image in the overlapping area on this group of contour corner points is calculated. The calibrated extrinsic parameters, i.e., position (x, y, z) and posture (roll, yaw, pitch) are repeatedly adjusted in small amplitudes to obtain new camera extrinsic parameters, and new mapping coordinate determinants and grayscale images are obtained using the new extrinsic parameters. The contour corner points captured previously are used to obtain the sum of the brightness ratios of the contour corner points under the extrinsic parameters, and compared with the original sum of the brightness ratios, the smaller one is a more accurate extrinsic parameter result.

[0026] The main advantages of this method are: using the determinant calculation model of OpenCV Eigen can obtain the determinant of the mapping result more efficiently and with less performance consumption. By grabbing a set of contour corner points to verify the accuracy, it can be free from the dependence on the calibration field and calibration plate. As long as there are contoured objects or light and shadow changes in the overlapping area of ​​the camera, a set of contour corner points can be obtained, reducing the requirements for the calibration environment. The accuracy is determined by the sum of the brightness differences of a set of contour corner points, and the verification of the external parameter calibration results is digitized, and the human eye is no longer required to confirm the calibration results of the external parameters. In addition, the system can automatically perform verification and correction multiple times. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the error stitching of the camera calibration external parameters;

[0028] Figure 2 This is a schematic diagram of the external parameter calibration of cameras A and B;

[0029] Figure 3 This is the calibration flow chart for the camera external parameter calibration results;

[0030] Figure 4 Schematic diagram of captured contour corner points;

[0031] Figure 5 This is the stitching effect diagram obtained by using a better solution of external parameters. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the embodiments.

[0033] Example 1: In the camera extrinsic calibration result correction method, the input parameters required by OpenCV Eigen mainly include the calibration extrinsic results of cameras A and B, the original image pixel determinants of images A and B and the calibration images of cameras A and B, such as Figure 2 shown.

[0034] like Figure 3 The camera extrinsic calibration result correction method based on OpenCV Eigen shown includes the following steps:

[0035] S1, obtaining camera extrinsic parameters and a stitching area camera image formed by calibrating images taken by multiple cameras with the camera extrinsic parameters, and generating a pixel point coordinate determinant according to the resolution of the stitching area camera image, wherein the pixel point coordinate determinant is used to represent the row and column position of each pixel point in the stitching area camera image in the image coordinate system;

[0036] S2. Convert the pixel points in the pixel point coordinate determinant from the image coordinate system to the camera coordinate system through the camera external parameter to obtain a mapping coordinate determinant;

[0037] S3, obtaining different camera mapping images under camera extrinsic configurations through the mapping coordinate determinant;

[0038] S4, converting the camera mapping image into a grayscale image by grayscale processing, and extracting contour corner points in the overlapping area of ​​the grayscale mapping images of different cameras by calculating the brightness gradient change to generate a feature point set of the overlapping area;

[0039] S5. For the feature point set in the overlapping area, calculate the brightness ratio difference in different camera images, and accumulate the brightness ratio difference to obtain a total as an evaluation index;

[0040] S6. Randomly perturb the position and attitude parameters of the camera extrinsic parameters to generate new camera extrinsic parameters, repeat steps S2 to S5, and use the new camera extrinsic parameters to recalculate the sum of brightness ratio differences;

[0041] S7. Select the camera external parameter corresponding to the lower sum of the brightness value ratios as the correction result.

[0042] Specifically, in S1, a pixel point determinant matrix is ​​created, and each element of the matrix is ​​initialized to 1; each pixel point of the image data is looped through, and the pixel point coordinate determinant is constructed by setting the first row and the second row of the determinant to be the column coordinate and the row coordinate of the pixel point, respectively.

[0043] In S2, the generated pixel point determinant and the camera extrinsic parameter determinant are received; the pixel point determinant and the camera extrinsic parameter determinant are matrix multiplied to obtain the camera mapping coordinate determinant.

[0044] In S4, the cvtColor function of the OpenCV library converts the image data captured by each camera into a grayscale image; defines a container for storing contour corner points; determines the overlapping area in the grayscale image, traverses each pixel point in the overlapping area, checks whether its coordinates are valid, and if valid, calculates the grayscale difference of the pixel point in the horizontal and vertical directions and the norm of the gradient vector, and stores the contour corner points whose gradient vector norm meets the requirements in the container.

[0045] In S5, the identified contour corner points are traversed and the brightness values ​​of the corresponding pixels in the grayscale image are accessed for calculation; for each contour corner point, its brightness value in two different camera images is obtained, and the ratio of the brightness values ​​is calculated; the sum of the squares of all the ratios is accumulated to obtain the sum of the ratios of the brightness values.

[0046] In S6, a random offset is generated to adjust six parameters of the camera extrinsic parameters, including: roll angle, pitch angle, yaw angle and three spatial position coordinates; an incremental matrix for parameter adjustment is generated; and the incremental matrix is ​​multiplied by the original camera extrinsic parameters to obtain new camera extrinsic parameters.

[0047] Perform multiple iterations of the camera extrinsic parameter adjustment and evaluation index calculation; record the evaluation index of each iteration; stop the iteration when the improvement of the evaluation index is less than the preset threshold. The amplitude of the camera extrinsic parameter adjustment is generated by a random distribution function, and the adjustment amplitude decreases with the increase of the number of iterations.

[0048] Embodiment 2: A camera extrinsic parameter correction system for implementing the method in Embodiment 1, comprising: a camera extrinsic parameter acquisition module, for acquiring the extrinsic parameters and image data of a camera; a pixel point determinant generation module, for generating a pixel point determinant according to the resolution of a camera image; an extrinsic parameter mapping module, for performing matrix multiplication of the pixel point determinant with the camera extrinsic parameter to generate a camera mapping coordinate determinant; a contour corner point extraction module, for converting a camera image into a grayscale image and identifying contour corner points in overlapping areas; an extrinsic parameter adjustment module, for generating new camera extrinsic parameters by offsetting six parameters of the camera extrinsic parameter; an evaluation calculation module, for calculating the brightness value of the contour corner point and generating a ratio sum; and a result output module, for outputting a corrected extrinsic parameter matrix.

[0049] Example 3: According to the resolution of the camera image, the coordinates of each pixel are written into OpenCV Eigen to form the determinant of the pixel. The process includes: (1) Matrix initialization, creating a matrix p_G of size 3×(rows×cols), and the initial value of all elements of the matrix is ​​1; (2) Nested loop traversal, the outer loop i traverses each row of the image, and the inner loop j traverses each column of the image. These two loops combined can traverse every pixel in the image; (3) Fill the matrix p_G: the 0th row is filled with the column index j of each pixel, the 1st row is filled with the row index i of each pixel, and the initial value of the 2nd row is 1 (specified when the matrix is ​​initialized). In this way, the element of the kth column in the matrix p_G is , corresponding to the pixel coordinates of the (i, j)th pixel in the image.

[0050] The pixel point determinant is multiplied by the camera's extrinsic parameter determinant to obtain the camera's extrinsic parameter mapping coordinate determinant, as provided by the following formula:

[0051] .

[0052] Through the above processing, the coordinate point determinant on the display screen corresponding to the image pixel point is obtained. The remap function of OpenCV is used to perform a table lookup operation through the determinant to map the display image corresponding to the camera extrinsic parameters.

[0053] Create a matrix ph to store the transformed point coordinates, define a 3×3 rotation matrix mR, multiply the rotation matrix mR with the matrix p_G, and store the result in the ph matrix.

[0054] Distortion correction: call the remap function, according to the mapping table p_GC_table_32F, and perform geometric transformation on the original image img, and remap it to the new image img_GC.

[0055] Use OpenCV to convert the camera image into a grayscale image represented only by brightness. In the overlapping area of ​​each camera image, the gradient change of brightness is calculated to determine whether the pixel is a contour corner point, and the contour corner points that meet the conditions are saved in a container. The processing flow is as follows: convert the color image into a grayscale image; obtain the pixel set of the first contour, loop through all the pixels in the pixel set, and make conditional judgments: skip the pixels close to the image boundary (the boundary width is 10 pixels) to prevent access out of bounds; calculate the grayscale value difference of the current pixel in the horizontal and vertical directions; use delta.norm() to calculate the Euclidean norm of the gradient vector (i.e., gradient strength), and filter the pixels according to the gradient strength: if the gradient strength is less than 15, skip the pixel (it is considered that the grayscale change is insufficient and not significant); add the pixels that meet the boundary conditions and gradient conditions to the pre-defined commonview.

[0056] like Figure 4 As shown, the red dots mark the contour corner points in the captured overlapping area that meet the filtering conditions.

[0057] In the grayscale image of the camera image in the overlapping area, the brightness values ​​of these pixels are taken out for comparison. The process is as follows: traverse the matrix storing the pixel coordinates; check the validity of the coordinate values; use the function getPixelValue to obtain the pixel value of a point on the image; normalize the pixel values ​​of the image and scale them using the normalization coefficient ncoef; calculate the difference in pixel values ​​of corresponding points in the two images and use the absolute value to find the difference; add the square of the error to the variable loss to measure the difference between the two images.

[0058] The following is a correction process of an extrinsic parameter adjustment to illustrate how to obtain new camera extrinsic parameters by offsetting the six parameters of the camera calibration extrinsic parameters. The specific process is as follows: define an array var of size 6 to store the current disturbance variables, corresponding to attitude (roll, pitch, yaw) and displacement (x, y, z) respectively; calculate the homogeneous matrix Tr of the original calibration extrinsic parameters according to the disturbance variable var, and generate a new transformation matrix deltaT based on random perturbations of attitude and displacement; during the disturbance process, each disturbance value is scaled with a corresponding resolution factor to control the size of the disturbance.

[0059] Map the new camera external parameters to obtain a new camera mapping image. Convert the new mapping image into a grayscale image, and extract the cached brightness value ratio of the contour corner position from the new grayscale image data to obtain the brightness ratio of the contour corner points in the overlapping area of ​​the new camera external parameter mapping.

[0060] Compare the sum of the brightness ratios of the original external parameters with the sum of the brightness ratios of the new external parameters. The external parameter corresponding to the lower value is the external parameter of the better solution. Figure 5 As shown in the figure, the stitching effect of the new camera external parameter mapping image in the overlapping area is compared with Figure 1 The stitching effect of the original camera extrinsic mapping images shown in the figure is more accurate in the overlapping areas.

[0061] The extrinsic parameter adjustment and correction process can be repeated multiple times, and more adjustments can be made through the camera extrinsic parameters to make the obtained camera extrinsic parameter results close to the optimal solution.

[0062] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A camera extrinsic calibration result correction method based on OpenCV Eigen, characterized in that: The steps include: S1, obtaining camera extrinsic parameters and a stitching area camera image formed by calibrating images taken by multiple cameras with the camera extrinsic parameters, and generating a pixel point coordinate determinant according to the resolution of the stitching area camera image, wherein the pixel point coordinate determinant is used to represent the row and column position of each pixel point in the stitching area camera image in the image coordinate system; S2. Convert the pixel points in the pixel point coordinate determinant from the image coordinate system to the camera coordinate system through the camera external parameter to obtain a mapping coordinate determinant; S3, obtaining different camera mapping images under camera extrinsic configurations through the mapping coordinate determinant; S4, converting the camera mapping image into a grayscale image by grayscale processing, and extracting contour corner points in the overlapping area of ​​the grayscale mapping images of different cameras by calculating the brightness gradient change to generate a feature point set of the overlapping area; S5, calculating the brightness ratio difference of the feature point set in the overlapping area in different camera images, and accumulating the total brightness ratio difference as an evaluation index; S6, randomly perturbing the position and attitude parameters of the camera extrinsic parameters to generate new camera extrinsic parameters, repeating steps S2 to S5, using the new camera extrinsic parameters to recalculate the sum of brightness ratio differences, iterating the parameter adjustment of the camera extrinsic parameters and the calculation of the evaluation index for multiple times, recording the evaluation index of each iteration, and stopping the iteration when the improvement of the evaluation index is less than a preset threshold; S7. Select the camera external parameter corresponding to the lower sum of the brightness value ratio differences as the correction result.

2. The camera extrinsic calibration result correction method based on OpenCV Eigen according to claim 1, characterized in that: The step S1 of generating the pixel coordinate determinant according to the resolution of the camera image in the stitching area includes: Initialize each element in the pixel determinant matrix to 1; Loop through the xy coordinates of each pixel in the camera image of the stitching area, and construct the pixel coordinate determinant by setting the first and second rows of the pixel determinant matrix to be the x coordinate and y coordinate of the pixel respectively, expressed as .

3. The camera extrinsic calibration result correction method based on OpenCV Eigen according to claim 1, characterized in that: The S2 includes: decomposing the original camera extrinsic parameter matrix into a rotation matrix and a displacement vector, wherein the rotation matrix is ​​used to describe the rotation relationship between the image coordinate system and the camera coordinate system, and the displacement vector represents the position of the camera relative to the global coordinate system; multiplying the pixel point coordinate determinant with the rotation matrix and adding the displacement vector to obtain the mapping coordinate determinant of the pixel point of the camera image corresponding to the display screen.

4. The camera extrinsic calibration result correction method based on OpenCV Eigen according to claim 3, characterized in that: The S3 includes: using the mapping coordinate determinant, using the remap function in the OpenCV library, and performing geometric correction on the converted image coordinates by a lookup table to achieve spatial calibration of the image, wherein the remap function interpolates and maps the pixel points in the image based on the mapping coordinate table of the camera extrinsic parameters, thereby generating a camera mapping image under the original camera extrinsic parameter matrix configuration.

5. The camera extrinsic calibration result correction method based on OpenCV Eigen according to claim 4, characterized in that: The S4 includes: Use the cvtColor function of the OpenCV library to convert each camera mapping image into a grayscale image; Define a container for storing contour corner points; Determine the overlapping area in the grayscale image, traverse each pixel in the overlapping area, check whether the coordinates of the pixel are valid, and if valid, calculate the grayscale difference of the pixel in the horizontal and vertical directions and the norm of the gradient vector, and store the contour corner points whose gradient vector norm meets the conditions in the container.

6. The method for correcting camera extrinsic calibration results based on OpenCV Eigen according to claim 5, characterized in that: The S5 includes: Traverse the identified contour corner points and calculate by accessing the brightness value of the corresponding pixel in the grayscale image; Get the brightness value of each contour corner point in two different camera images and calculate the ratio of the brightness values; The sum of the squares of all the ratios is accumulated to obtain the sum of the ratios of the brightness values.

7. The method for correcting camera extrinsic calibration results based on OpenCV Eigen according to claim 6, characterized in that: The S6 includes: Generate random offsets to adjust six parameters of the camera's extrinsic parameters, including roll angle, pitch angle, yaw angle, and three spatial position coordinates; Generate an incremental matrix for parameter adjustment; The incremental matrix is ​​multiplied by the original camera extrinsic parameters to obtain new camera extrinsic parameters.

8. The method for correcting camera extrinsic calibration results based on OpenCV Eigen according to claim 7, characterized in that: Perform multiple iterations on the parameter adjustment of the camera extrinsic parameters and the calculation of the sum of the brightness ratio differences; Record the sum of the calculated brightness ratio differences after each iteration; The stitching effect of the overlapping area is optimized by minimizing the sum of the brightness ratio differences until the optimal camera extrinsic parameters are obtained.

9. The method for correcting camera extrinsic calibration results based on OpenCV Eigen according to claim 8, characterized in that: The amplitude of the camera extrinsic parameter adjustment is generated by a random distribution function, and the adjustment amplitude decreases as the number of iterations increases.

10. A camera extrinsic parameter calibration system for implementing the camera extrinsic parameter calibration result correction method based on OpenCV Eigen as claimed in claim 1, characterized in that: include: The camera external parameter acquisition module is used to obtain the camera's external parameters and image data; A pixel point determinant generation module, used to generate a pixel point determinant according to the resolution of the camera image; An extrinsic parameter mapping module, used to perform matrix multiplication on the pixel point determinant and the camera extrinsic parameter to generate a mapping coordinate determinant; A contour corner point extraction module is used to convert the camera image into a grayscale image and identify the contour corner points in the overlapping area; The extrinsic parameter adjustment module is used to generate new camera extrinsic parameters by offsetting six parameters of the camera extrinsic parameters; An evaluation calculation module, used for calculating the brightness value of the contour corner points and generating a ratio sum; The result output module is used to output the corrected external parameter matrix.

Images