Camera calibration method, storage medium and electronic device
The camera eccentricity correction is performed by the feature points in the external pattern, the camera focal length and distortion coefficient, which solves the problem of inaccurate calibration of the camera's external parameters, and achieves the improvement of camera performance and image quality.
Patent Information
- Application Number
- CN202111670279.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2041-12-31
AI Technical Summary
During the production process of existing cameras, the eccentric changes of the camera are difficult to verify, resulting in inaccurate calibration of the external parameters of the camera, affecting the effect of the camera function, especially in the panoramic stitching function, which may lead to image distortion.
The camera external parameters are calibrated by feature points in the external pattern, and the camera focal length and distortion coefficient are arranged by preset rules for distortion correction, the actual eccentricity of the camera is calculated, and the external parameters are adjusted through optimization methods to ensure accurate calibration.
It realizes automatic correction of camera eccentricity during calibration of external parameters, ensures good camera performance, reduces image distortion, and improves the accuracy of camera functions.
Smart Images
Figure CN114299164B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to camera calibration and verification, and in particular to a camera calibration method, a storage medium and an electronic device. Background Art
[0002] The on-board camera is the core sensor in the field of automotive driver assistance systems (ADAS) and autonomous driving. Most application scenarios require the calibration of the camera's internal and external parameters. Usually, the internal parameters (such as camera focal length, camera eccentricity, and camera distortion coefficient) are calibrated on the camera production line. In the vehicle factory, after the camera is installed, the camera's external parameters (installation position and installation posture) are calibrated. The calibration of external parameters depends on the accuracy of internal parameters. From the perspective of engineering practice, eccentricity is the most likely to change during the camera production process. Many camera manufacturers only provide theoretical internal parameters. Even if some camera manufacturers have a process for calibrating internal parameters during the production process, the accuracy of internal parameters, especially eccentricity, is difficult to verify. Cameras with poor eccentricity will have inaccurate calibrated external parameters due to the large difference between the actual eccentricity and the eccentricity provided to the calibration algorithm, which will reduce the effectiveness or even make many camera-related functions ineffective. For example, in the panoramic stitching function, if the eccentricity does not meet a certain accuracy, the stitched image will be distorted (such as Figure 2 shown). Summary of the invention
[0003] The object of the present invention is to provide a camera calibration method, a storage medium and an electronic device for automatically correcting camera eccentricity during the process of calibrating external parameters.
[0004] A camera calibration method is provided, which calibrates camera extrinsic parameters by means of an external pattern, wherein the pattern includes a plurality of feature points, and the plurality of feature points are arranged according to a preset rule and distributed on at least one straight line. The camera calibration method includes the steps of:
[0005] S10, capturing an image through a camera, identifying feature points in the captured image, confirming the arrangement order and world coordinates of the feature points, and obtaining a mapping relationship between the image coordinates and the world coordinates of the feature points;
[0006] S20, finding the feature points actually arranged on a straight line according to the arrangement order and the world coordinates, and determining the sequence of the feature points on each straight line;
[0007] S30, performing distortion correction on the image coordinates of the feature points according to the camera focal length, the initial eccentricity value and the distortion coefficient to obtain corrected image coordinates of the feature points;
[0008] S40, constructing a metric function for confirming the degree of distortion of a straight line using the corrected feature point image coordinates, and calculating the actual eccentricity of the camera according to the metric function;
[0009] S50, re-distortion-correcting the feature points in the image captured in step S10 according to the actual eccentricity, the camera focal length and the distortion coefficient to obtain new corrected feature point image coordinates, and then calculating the camera extrinsic parameters using the new corrected feature point image coordinates and the corresponding world coordinates;
[0010] S60, evaluating the external parameters, confirming that the calibration is successful and outputting the external parameters if they meet the requirements, otherwise outputting the evaluation results including the accuracy value.
[0011] As an implementation manner, when the camera is a vehicle-mounted camera, the external pattern is horizontally arranged on both sides or around the parking space.
[0012] As an implementation mode, when a plurality of feature points are arranged according to a preset rule and distributed on two or more straight lines, these straight lines intersect or are parallel, and when they intersect, the intersection point is one of the plurality of feature points.
[0013] As an implementation mode, the metric function includes the focal length and distortion coefficient of the camera and the eccentricity to be confirmed, and the eccentricity when the metric function is minimized by using an optimization method is the actual eccentricity.
[0014] A storage medium includes instructions, wherein the instructions are used to implement the above-mentioned camera calibration method.
[0015] An electronic device comprises a processor and the above-mentioned storage medium, wherein the electronic device calls instructions of the storage medium through the processor to implement the above-mentioned camera calibration method.
[0016] The camera calibration method of the present invention can automatically correct the camera eccentricity when calibrating external parameters, and use the corrected eccentricity to calibrate the external parameters, so that the external parameters are accurate and the camera performance is ensured to be good. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A schematic diagram of an image captured by a camera in a camera calibration method according to an embodiment.
[0018] Figure 2 When the camera eccentricity is not corrected, the calibrated external parameters are used to capture images and stitch them into a bird's-eye view.
[0019] Figure 3 In order to automatically correct the eccentricity in one embodiment of the present invention, images are taken using calibrated external parameters and stitched into a bird's-eye view. DETAILED DESCRIPTION
[0020] The camera calibration method of the present invention will be further described in detail below in conjunction with specific embodiments and drawings.
[0021] In the camera calibration method of the present invention, an external image is used to calibrate the external parameters of the camera. The external image can be printed, sprayed or applied on a movable plate or sheet or on the ground or wall, or can be displayed on a display. The external image should include multiple feature points, which should be arranged according to a preset rule and distributed on at least one straight line, and when distributed on two or more straight lines, part of these straight lines are preferably intersecting or perpendicular to each other, and part of them are parallel to each other. In this embodiment, the camera is a vehicle-mounted camera, and at least four cameras on each vehicle need to be calibrated. The location where the vehicle is parked is defined as a parking space, and the external image is preferably horizontally set on both sides or around the parking space. Multiple feature points are scattered on both sides or around the parking space, and the at least one straight line is preferably perpendicular or parallel to the longitudinal axis of the vehicle body (parallel or perpendicular to the transverse axis of the vehicle body). In this embodiment, the external image uses a black and white chessboard with black squares and white squares arranged alternately, and the four corners of the black and white squares are used as feature points, and these feature points are respectively distributed on multiple straight lines that are parallel or perpendicular to each other. More feature points define more straight lines, and the span between parallel straight lines is larger. More perpendicular straight lines have a better correction effect on eccentricity. In other embodiments, the external image may include a plurality of rectangular QR codes arranged in rows and columns, and the corner points of the QR codes are used as feature points. In this way, a site can be realized where all vehicle-mounted cameras can be calibrated and verified without turning around.
[0022] The camera calibration method of the present invention is used to determine the external parameters of a camera, and the instructions for implementing the method are stored in a storage medium. The storage medium can be fixed in the electronic system of the car, or it can be a pluggable storage medium. The user can call the instructions of the storage medium through a processor of an electronic device to implement the above-mentioned camera calibration method. For example, when the camera calibration method of the present invention is used to calibrate and verify the external parameters of a vehicle-mounted camera, the electronic device can be a vehicle-mounted host or an external control host connected to the vehicle-mounted host, which runs the camera calibration method of the present invention by calling instructions to calibrate and verify the external parameters of all vehicle-mounted cameras.
[0023] During the camera calibration process, the following coordinate systems are used: (1) World coordinate system, which is a coordinate system used to describe the camera position coordinates. For example, a world coordinate system is established using a calibration plate or a vehicle coordinate system. In this embodiment, the world coordinate system is established using the vehicle body. The longitudinal axis of the vehicle body is the Xw axis, with the forward direction being positive, the transverse axis of the vehicle body is the Yw axis, with the left side being positive, and the Zw axis is vertically upward, satisfying the right-hand coordinate system; (2) Camera coordinate system, with the optical center as the origin and the optical axis as the Zc axis. Xc and Yc are parallel to the x-axis and y-axis of the imaging plane, respectively; (3) Image coordinate system, a 2D coordinate system of the imaging plane. The origin is the intersection of the optical axis and the imaging plane. The x-axis and y-axis are parallel to the Xc axis and Yc axis of the camera coordinate system, and the optical center is the midpoint of the image; (4) Pixel coordinate system, which finally presents the image in front of us. The upper left corner is the origin, and the two axes are parallel to the image coordinate system, respectively.
[0024] The camera calibration method of the present invention is described below with a specific embodiment, and the method includes steps S10 to S60.
[0025] In step S10, a camera is used to capture an image, identify feature points in the captured image, and confirm the arrangement order and world coordinates of these feature points. Figure 1 In the image captured, 20 feature points can be identified. The arrangement rules of these feature points can be compared with the actual external pattern to confirm their correspondence with the actual feature points (feature points in the physical world), and their arrangement order and world coordinates can be determined. In a fixed place, the world coordinates of each actual feature point can be measured before calibration and then stored in the storage medium together with the arrangement order. In this way, the mapping relationship between the image coordinates and the world coordinates of the feature points is confirmed.
[0026] Step S20, find the feature points actually arranged on a straight line in the image according to the arrangement order and the world coordinates, and determine the sequence of these feature points. Figure 1 In the figure, the four feature points numbered 0, 1, 10, and 11 are on a straight line in the world coordinates and are arranged from left to right. The corner points numbered 3, 2, 4, 14, 13, and 12 are also on a straight line in the world coordinates and are arranged from left to right. The corner points numbered 0, 3, 7, and 9 are also on a straight line in the world coordinates and are arranged from front to back on the left side of the car. In this way, all feature point sequences on a straight line are determined in sequence.
[0027] S30, according to the camera principle, in the absence of distortion, a straight line in the real world is still a straight line in the image, therefore, the image coordinates of the feature points in the image coordinate system are subjected to distortion correction according to the camera focal length (fx, fy), the initial eccentricity value and the distortion coefficient (k1, k2, p1, p2, k3, k4, k5, k6) to obtain the corrected image coordinates of the feature points. In the corrected image, the line connecting the feature points belonging to the same straight line is still not a straight line, although the curvature becomes smaller, because the initial eccentricity value of the camera is inaccurate.
[0028] S40, using the corrected feature point image coordinates to construct a metric function for confirming the degree of straight line distortion, and calculating the actual eccentricity (cx1, cy1) based on the metric function. These metric functions include known focal length and distortion coefficient and unknown eccentricity (cx, cy). The eccentricity when the metric function is minimized by the optimization method is the actual eccentricity (cx1, cy1). Without loss of generality, the example solution is as follows:
[0029] Assuming there is only one straight line, the straight line formed by the sequence of correction points (corrected feature points) under a certain eccentric combination (cx0, cy0) can be defined as:
[0030] Where x0 and x1 represent the horizontal and vertical coordinates of the correction point respectively;
[0031] Construct the metric function:
[0032] Where m represents the number of feature points on this line, h θ (x i ) means that the coordinates of each point Substitute into formula (1) h(θ);
[0033] Use general optimization iterations, such as gradient descent, or directly use a simple least squares method to find the minimum J(θ):
[0034] Derivative J(θ) with respect to θ,
[0035]
[0036] Combined with formula (1), formula (3) is transformed into the following:
[0037]
[0038] The optimized solution is Obtained, so there is:
[0039]
[0040] Transforming formula (5), we get:
[0041]
[0042] Formula (6) is a set of linear equations in two variables. We can solve for θ0 and θ1, and then substitute θ0 and θ1 into formula (2) to get the minimum value. So for a given (cx0, cy0), J(θ) has the minimum value J(θ 0 (cx0, cy0)). Construct the function G(cx, cy) = J(θ 0 (cx, cy)), find a certain eccentric combination (cx1, cv1) that minimizes G(cx, cy), that is, solve Then the eccentric combination (cx1, cy1) is considered to be an eccentric value that meets the conditions (the simplest method is, for example, traversal: (cx, cy) can be traversed within a certain range, and each (cx, cy) has a corresponding minimum value J(θ(cx, cy)), and the (cx, cy) corresponding to the minimum value of these J(θ(cx, cy)) is the required (cx1, cy1)).
[0043] When there are multiple straight lines, the variance or mean of the relatively good metric function is used to determine the final actual eccentricity. For example, when there are two straight lines, when the eccentricity is (cx2, cy2), the metric functions are 0.2 and 0.3 respectively; when the eccentricity is (cx3, cy3), the metric functions are 0.25 and 0.2 respectively. Since 0.25+0.2 is less than 0.2+0.3, the eccentricity (cx3, cy3) is determined as the final actual eccentricity.
[0044] S50, according to the actual eccentricity (cx1, cy1), camera focal length (fx, fy), distortion coefficient (k1, k2, p1, p2, k3, k4, k5, k6), the feature point image coordinates in the original image are re-distorted to obtain new corrected feature point image coordinates, and then the camera extrinsic parameters are calculated using the mapping relationship between the new corrected feature point image coordinates and the corresponding world coordinates. The optional algorithms for calculating the camera extrinsic parameters are relatively mature, such as but not limited to Zhang Zhengyou's camera calibration calculation method, which will not be repeated here.
[0045] S60, evaluating the external parameter, if it meets the requirements, confirming the calibration is successful and outputting the external parameter, otherwise outputting the evaluation result including the accuracy value. The method of evaluating the external parameter is very mature and will not be described here.
[0046] In this way, the camera eccentricity parameters can be corrected while calibrating the camera external parameters. The calibrated external parameters are more accurate, so the output image distortion is small. For example, Figure 2When the camera eccentricity is not corrected, the bird's-eye view obtained by taking images using the calibrated external parameters and stitching them together will have distortions on the left and right sides of the image. Figure 3 In order to automatically correct the eccentricity in one embodiment of the present invention, images are taken using calibrated external parameters and stitched into a bird's-eye view. It can be seen that each area is normal after stitching, and no obvious distortion can be observed by the naked eye.
[0047] In the above embodiments, the characteristic points are distributed on straight lines that are parallel or perpendicular to each other. It can be understood that in other embodiments, the angle between these straight lines may be less than 90 degrees.
[0048] Although the present invention is described in conjunction with the above specific embodiments, it is obvious that those skilled in the art can make many substitutions, modifications and changes according to the above content. Therefore, all such substitutions, improvements and changes are included in the spirit and scope of the appended claims.
Claims
1. A camera calibration method, which calibrates camera extrinsic parameters by means of an external pattern, wherein the pattern includes a plurality of feature points, which are arranged according to a preset rule and distributed on at least one straight line, characterized in that: The camera calibration method comprises the steps of: S10, capturing an image through a camera, identifying feature points in the captured image, confirming the arrangement order and world coordinates of the feature points, and obtaining a mapping relationship between the image coordinates and the world coordinates of the feature points; S20, finding the feature points actually arranged on a straight line according to the arrangement order and the world coordinates, and determining the sequence of the feature points on each straight line; S30, performing distortion correction on the image coordinates of the feature points according to the camera focal length, the initial eccentricity value and the distortion coefficient to obtain corrected image coordinates of the feature points; S40, constructing a metric function for confirming the degree of distortion of a straight line using the corrected feature point image coordinates, and calculating the actual eccentricity of the camera according to the metric function; wherein the metric function includes the focal length and distortion coefficient of the camera and the eccentricity to be confirmed, and the eccentricity when the metric function is minimized by using an optimization method is the actual eccentricity; S50, re-distortion-correcting the feature points in the image captured in step S10 according to the actual eccentricity, camera focal length and distortion coefficient to obtain new corrected feature point image coordinates, and then calculating the camera extrinsic parameters using the new corrected feature point image coordinates and corresponding world coordinates as well as the actual eccentricity and focal length; S60, evaluating the external parameters, confirming that the calibration is successful and outputting the external parameters if they meet the requirements, otherwise outputting the evaluation results including the accuracy value.
2. The camera calibration method according to claim 1, characterized in that: When the camera is a vehicle-mounted camera, the external patterns are horizontally arranged on both sides or around the parking space.
3. The camera calibration method according to claim 1, characterized in that: When multiple feature points are arranged according to a preset rule and distributed on two or more straight lines, these straight lines intersect or are parallel. When they intersect, the intersection point is one of the multiple feature points.
4. A storage medium, characterized in that: The storage medium includes instructions, and the instructions are used to implement the camera calibration method as described in any one of claims 1 to 3.
5. An electronic device, characterized in that: The electronic device comprises a processor and a storage medium as claimed in claim 4, and the electronic device calls instructions of the storage medium through the processor to implement the camera calibration method as claimed in any one of claims 1 to 3.