Joint calibration method and system for automobile panoramic camera

Abstract

The invention discloses an automobile panoramic camera joint calibration method and system. The method comprises two steps of primary calibration and global optimization. According to the primary calibration, calibration cloth with the same number as the cameras is arranged on the ground, based on the plane homography transformation principle, homography matrixes of all the cameras are sequentially solved in an iteration mode, and internal parameters of the cameras are not needed in the whole process. In the global optimization, an initial calibration result is used as an initial value, minimization of re-projection errors of all angular points is taken as a target, and angular point world coordinates of all calibration cloth and homography matrixes of all cameras are synchronously optimized by adopting a nonlinear optimization algorithm, so that accumulated errors caused by chain transmission are effectively eliminated. The invention further protects the electronic equipment for executing the method, a computer storage medium and an automobile panoramic imaging system comprising the method. According to the method, the calibration process is simplified, the requirement for hardware knowledge is reduced, the calibration precision and robustness are remarkably improved, and meanwhile, a comprehensive patent protection system is constructed.

Description

Technical Field

[0001] This invention relates to the fields of automotive driver assistance and image processing technology, specifically to a camera joint calibration method for an automotive panoramic surround view system, an electronic device for performing the method, and a panoramic imaging system incorporating the calibration method. Background Technology

[0002] A car surround-view system acquires environmental images through multiple cameras installed around the vehicle and stitches these images together to create a panoramic view, providing the driver with an overall perspective of the vehicle's surroundings. A prerequisite for achieving high-quality image stitching is precise joint calibration of all cameras.

[0003] Currently, mainstream panoramic calibration methods mainly suffer from the following two types of problems: The first type of method requires laying a huge and complex calibration cloth and precise measurement of the physical dimensions of the target. This method usually requires knowledge of the camera's internal parameters, and the entire calibration process is complex, inefficient, and has high site requirements, making it inconvenient for rapid deployment.

[0004] While the second type of method simplifies the setup process by using multiple small calibration cloths or calibration boards, its calibration process still relies on accurate camera intrinsic parameters and involves calculating the rotation matrix (R) and translation vector (t) between the camera and the calibration board. This presents limitations for certain lenses (such as fisheye lenses) because some calibration methods for these lenses cannot calculate the complete intrinsic parameters. Therefore, when the camera intrinsic parameters are unknown or inaccurately estimated, using traditional methods will lead to calibration failure or poor results.

[0005] Furthermore, existing calibration methods generally employ a chain-like calculation sequence when solving for parameters from multiple cameras. This approach lacks a global adjustment step, causing errors to accumulate during the propagation process, ultimately affecting the accuracy and visual quality of image stitching. Summary of the Invention

[0006] Purpose of the invention The purpose of this invention is to overcome the shortcomings of the prior art and provide a joint calibration method and system for automotive panoramic cameras that has a simple calibration process, does not rely on the precise intrinsic parameters of the camera, and can effectively suppress cumulative errors.

[0007] Technical solution To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a joint calibration method for automotive panoramic cameras, comprising the following steps: Initial calibration steps: S1. Arrange calibration cloths: Based on the positions of N cameras on the vehicle body, arrange N calibration cloths on the ground plane, where N≥4. Each calibration cloth is located within the common field of view of a pair of adjacent cameras and ensures that it is fully displayed in the image of the corresponding camera. The calibration cloth is preferably square, but its shape is not limited to this; it can also be rectangular or any pattern with at least three identifiable feature points.

[0008] S2. Acquisition and Coordinate Extraction: Simultaneously acquire images from all cameras and extract the pixel coordinates of the corresponding calibrated corner points or feature points in each image.

[0009] S3. Iteratively solve the homography matrix: Using one of the calibration cloths as the origin of the world coordinate system, initialize the world coordinates of its corner points; starting from the camera corresponding to the initial calibration cloth, calculate the homography matrix of each camera in sequence, and use the current matrix to solve the world coordinates of the next adjacent calibration cloth through inverse transformation; iterate this process until the homography matrix of all cameras is calculated.

[0010] Global optimization steps: S4. Parameter Initialization: The world coordinates of each calibration corner point and the homography matrix of each camera, obtained in the initial calibration step, are used as the initial values ​​for global optimization. Specifically, this includes: Corner coordinate assignment: Select one corner of the initial calibration cloth as the origin of the coordinate system and fix it (e.g., set it to (0,0)), and fix one of its adjacent corners to provide scale constraints (e.g., set it to (L,0), where L is the physical side length of the calibration cloth); use the world coordinates of all other corners as parameters to be optimized, and assign the coordinates obtained from the initial calibration to these parameters as initial values.

[0011] Homography matrix assignment: First, invert the homography matrix of all cameras, then normalize it, and then use the normalized matrix parameters as the initial values ​​of the homography matrix parameters to be optimized.

[0012] S5. Optimization Problem: The optimization objective is to minimize the sum or square of the absolute values ​​of the differences between the pixel coordinates formed by reprojecting all calibrated corner points onto the imaging plane of each camera through the currently estimated homography matrix and the actual pixel coordinates of the corner points.

[0013] S6. Optimization: A nonlinear optimization algorithm, preferably the Levenberg-Marquardt algorithm, is used to solve the optimization problem, and the world coordinates of all calibrated corner points and the homography matrix parameters of all cameras are optimized simultaneously.

[0014] In a second aspect, the present invention provides a panoramic imaging system for automobiles, comprising: N cameras are installed on the vehicle body, where N≥4; The processing unit is configured to perform the following operations: Perform the method described in any of the first aspects to calibrate the homography matrix of all cameras to the ground plane; Receive real-time images from each camera; Using the homography matrix obtained from the calibration, the real-time images are transformed to a unified ground plane coordinate system and then stitched together to generate a panoramic image.

[0015] Preferably, the processing unit is further configured to map the stitched panoramic image onto a three-dimensional vehicle model to generate a three-dimensional panoramic view.

[0016] Beneficial effects Compared with the prior art, the technical solution of the present invention has the following significant advantages: 1. Simplified process, independent of intrinsic parameters: This invention uses multiple small calibration cloths, avoiding the hassle of laying complex large calibration objects. The entire calibration process is based on the homography transformation between the imaging plane and the ground plane, requiring no knowledge of the camera's internal parameters (especially suitable for fisheye lenses), and no calculation of rotation matrices and translation vectors. This greatly reduces the requirements for hardware and prior knowledge, making the calibration process simpler and more universal.

[0017] 2. Effective Suppression of Cumulative Errors: After the initial calibration, this invention introduces a global optimization step. This step uses bundle adjustment to jointly adjust all corner coordinates and all homography matrices within the same optimization framework, minimizing global reprojection errors. This effectively solves the cumulative error problem caused by chain propagation, significantly improving calibration accuracy and the quality of subsequent image stitching.

[0018] 3. High application flexibility: The method has more relaxed requirements on the shape of the calibration cloth and provides different optimization objective functions and algorithm selections, making it adaptable to different application scenarios and accuracy requirements. Attached Figure Description

[0019] Figure 1 This is a top view diagram showing the arrangement of the calibration cloth and camera positions in an embodiment of the present invention.

[0020] Figure 2 This is a flowchart of the initial calibration steps of the present invention.

[0021] Figure 3 This is a schematic diagram of the world coordinates of the corner point of the calibration cloth A in the initial calibration step of this invention.

[0022] Figure 4 This is a flowchart of the global optimization steps of the present invention.

[0023] Figure 5This is a schematic diagram of a vehicle panoramic imaging system according to an embodiment of the present invention. Detailed Implementation

[0024] To further illustrate the various embodiments, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, primarily used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementations and the advantages of the present invention.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] Initial calibration example: Take a car equipped with four cameras (front, rear, left, and right) as an example.

[0027] 1. For example Figure 1 As shown, four square calibration cloths with side length L are placed on the ground around the vehicle, labeled A, B, C, and D respectively. Each calibration cloth is located in the overlapping field of view of two adjacent cameras. For example, calibration cloth A is located in the overlapping field of view of the front camera and the left camera, calibration cloth B is located in the overlapping field of view of the front camera and the right camera, and so on.

[0028] 2. Simultaneously trigger four cameras to acquire images.

[0029] 3. In the computer, extract the pixel coordinates of the four corner points of each calibration cloth in all images in a preset order.

[0030] 4. Perform iterative solutions, such as... Figure 2 As shown: a. Assume the previous camera and its corresponding calibration cloth (calibration cloth A) are the starting point. Initialize the world coordinates of the four corner points of calibration cloth A as (0,0), (L,0), (L,L), (0,L). Figure 3 As shown.

[0031] b. Calculate the homography matrix H1 of the front camera based on the corner pixel coordinates of calibration cloth A in the front camera image. Since the world coordinates of calibration cloth A have been initialized and its pixel coordinates in the front camera image are known, these two sets of two-dimensional point pairs jointly constrain a homography transformation from the world plane to the image plane. By solving this transformation, the homography matrix H1 that accurately describes the projection relationship of the front camera can be obtained.

[0032] c. Using the inverse transformation of H1, the world coordinates of calibration cloth B are calculated based on the corner pixel coordinates of the adjacent right-side calibration cloth (calibration cloth B) in the front camera image. Since the same camera (front camera) simultaneously observes calibration cloth A and calibration cloth B, they are located on the same ground plane and share the same camera projection model. Therefore, the obtained H1 is also applicable to the projection process of calibration cloth B in this image. By applying the inverse transformation of H1, the pixel coordinates of calibration cloth B can be accurately extrapolated to the world coordinate system.

[0033] d. Next, select the right camera, and calculate the homography matrix H2 of the right camera based on the corner pixel coordinates of the calibration cloth B in its image and the world coordinates just obtained.

[0034] e. Similarly, the world coordinates of the rear right calibration cloth (calibration cloth C) are solved by using the inverse transformation of H2, and then the homography matrix H3 of the rear camera is calculated.

[0035] f. Finally, use H3 to calculate the world coordinates of the calibration cloth (calibration cloth D) on the left rear side, and finally calculate the homography matrix H4 of the left camera. At this point, the initial calibration is complete.

[0036] At this point, we have solved for the coordinates of the corner pixels of the four calibration plotters in the world coordinate system, as well as the homography matrices H1, H2, H3, and H4 from the four cameras to the ground plane in the world coordinate system. With these solved parameters, we can now achieve panoramic stitching on the ground plane.

[0037] However, it is not accurate enough because the solution process for our homography matrix and the coordinates of the calibration cloth in the world coordinate system depends on the previous measurement data. Due to the errors in each step, the errors will become larger and larger (i.e., the error accumulation problem). When we continue to calculate the corner pixel coordinates of the calibration cloth A through the homography matrix H4, we will find that the coordinates do not coincide with the coordinates at the time of initialization.

[0038] Global optimization example: like Figure 4 As shown, to eliminate accumulated errors and improve the coordinate accuracy of multiple calibration cloths and the accuracy of the homography matrix from the imaging surface of each camera to the ground plane, the following global optimization is performed after the initial calibration is completed: Parameter initialization: Corner coordinates: The first corner (0,0) and the second corner (L,0) of the calibration cloth A are fixed to provide overall dimensional constraints. The coordinates of the remaining 14 corner points (a total of 28 parameters) are to be optimized, and the initial values ​​are the coordinates obtained from the initial calibration.

[0039] Homography matrix: Invert and normalize H1, H2, H3, and H4 respectively to obtain a total of 32 parameters of the four matrices, which are used as the initial values ​​of the homography matrix to be optimized.

[0040] Cost function construction: The cost function is the sum of the absolute differences between the pixel coordinates of all 16 corner points (total 16) after reprojection onto the imaging surfaces of the four cameras and the actual extracted pixel coordinates. Its formula is expressed as:

[0041] X and Y represent the X and Y coordinates of the corner point to be optimized, respectively; H represents the four homography matrices to be optimized. These represent the projected coordinates of the corner point on the camera's imaging plane, and the reprojected coordinates of the corner point on the camera's imaging plane, obtained by solving the estimated world coordinates and the estimated homography matrix of the corner point, respectively.

[0042] Optimization: The Levenberg-Marquardt algorithm is used to minimize the above cost function, while optimizing and iterating on 60 parameters (28 corner coordinates + 32 homography matrix parameters) until convergence.

[0043] Finally, the four homography matrices obtained after optimization are the high-precision final calibration parameters that have eliminated cumulative errors and can be used for subsequent panoramic image stitching.

[0044] System application examples: Reference Figure 5 The automotive panoramic imaging system 700 includes multiple cameras 701 (e.g., front, rear, left, and right), a processing unit 702 (e.g., a vehicle infotainment chip or a dedicated image processing module), and a display unit 703. The processing unit 702 executes a stored program to first calibrate the cameras 701 using the method of this invention, obtaining a high-precision homography matrix. In the real-time panoramic function, the processing unit 702 receives images from various sources, performs image transformation and stitching using calibration parameters, and finally outputs a seamless panoramic view or 3D panoramic view on the display unit 703.

[0045] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for joint calibration of automotive panoramic camera, characterized in that, The method comprises the following steps: A primary calibration step: N calibration cloths are arranged on the ground plane, N is equal to the number of vehicle body cameras and N≥4, and each calibration cloth is located in the common field of view of a pair of adjacent cameras; Images of all cameras are synchronously captured, and pixel coordinates of corresponding corner points of the calibration cloths in the images are extracted; A world coordinate system is initialized with one of the calibration cloths as the origin, and the world coordinates of the corner points of the calibration cloth are initialized; starting from the camera corresponding to the initial calibration cloth, the homography matrix of each camera is sequentially calculated, and the world coordinates of the next adjacent calibration cloth are solved by inverse transformation using the current matrix; the process is iteratively executed until the homography matrix of the Nth camera is calculated; A global optimization step: The world coordinates of the corner points of the calibration cloths and the homography matrices of the cameras solved in the primary calibration step are used as initial values for global optimization; An optimization problem is constructed by minimizing the sum of absolute values of differences between the pixel coordinates of all calibration cloth corner points reprojected onto the imaging planes of the cameras and the actual pixel coordinates of the corner points; A nonlinear optimization algorithm is used to solve the optimization problem, and the world coordinates of all calibration cloth corner points and the homography matrix parameters of all cameras are simultaneously optimized.

2. The automobile panoramic camera joint calibration method according to claim 1, characterized in that, In the global optimization step, when the world coordinates of the corner points to be optimized are initially assigned, the coordinates of two adjacent corner points of the initial calibration cloth are fixed to provide scale and origin constraints, and the world coordinates of the remaining corner points are used as optimization parameters.

3. The method of claim 2, wherein, The two adjacent corner points of the initial calibration cloth are fixed to provide scale and origin constraints by selecting one corner point of the initial calibration cloth as the origin of the coordinate system and setting it as (0, 0), and fixing one adjacent corner point and setting it as (L, 0), where L is the physical side length of the calibration cloth.

4. The method of claim 1, wherein, In the global optimization step, when the homography matrix parameters to be optimized are initially assigned, the parameter values of the homography matrix obtained in the primary calibration step after inverse and normalization processing are used as initial values.

5. The method of claim 1, wherein, The nonlinear optimization algorithm is the Levenberg-Marquardt algorithm.

6. The method of claim 1, wherein, The optimization objective is to minimize the sum of squares of all corner point re-projection errors.

7. The method of claim 1, wherein, The shape of the calibration cloth is a square, a rectangle, or any shape with at least three identifiable feature points. 8.The automobile panoramic camera joint calibration method according to claim 1, characterized in that, The camera is a fisheye camera, and its internal parameters and distortion coefficients do not need to be provided during the entire calibration process.

9. An automotive panoramic imaging system, characterized by, It comprises: N cameras mounted on a vehicle body, N≥4; A processing unit configured to perform the following operations: Perform the method of any one of claims 1-8 to calibrate the homography matrix of all cameras to the ground plane; Receive real-time images of each camera; Use the homography matrix obtained by calibration to transform the real-time images to a unified ground plane coordinate system and stitch them to generate a panoramic image.

10. The automotive panoramic imaging system of claim 9, wherein, The processing unit is further configured to map the stitched panoramic image onto a three-dimensional vehicle model to generate a three-dimensional panoramic view.

Images