Multi-camera joint calibration method and device, electronic equipment and storage medium

Abstract

The application provides a multi-camera joint calibration method, device, electronic equipment and storage medium. The method comprises: calibrating the camera according to a plurality of inner parameter calibration board images collected by the camera to obtain an inner parameter matrix of the camera; and calibrating the camera according to a plurality of outer parameter calibration board images collected by the camera and the inner parameter matrix to obtain an outer parameter matrix of the camera, thereby completing the calibration of the camera; wherein the inner parameter calibration board images and the outer parameter calibration board images are both checkerboard images; the above calibration step of the camera is repeatedly performed multiple times in the same time period to obtain the inner parameter matrix and the outer parameter matrix of each camera in a plurality of cameras, thereby completing the calibration of the plurality of cameras. The method can solve the problem of how to calibrate a plurality of cameras to improve the camera calibration efficiency.

Description

Technical Field

[0001] This application relates to camera calibration technology in intelligent driving, and more particularly to a multi-camera joint calibration method, device, electronic device, and storage medium. Background Technology

[0002] The most important functions of intelligent driving include road assist, lane changing, and traffic jam assist. These functions require sensors to comprehensively identify the environment and targets, and make optimal decisions based on the identification results. For example, in long-haul logistics scenarios, when intelligent trucks are driving intelligently, the sensors installed on the vehicle need to comprehensively identify the environment and targets, and make optimal driving decisions accordingly to ensure that goods can be transported safely and quickly to their destination.

[0003] Sensors are the primary means by which intelligent driving systems acquire environmental and target information. Sensors include millimeter-wave radar, lidar, cameras, and inertial measurement units. To ensure the accuracy and reliability of the data acquired by sensors, calibration is necessary. Taking cameras as an example, calibration determines the internal parameters of the cameras (such as focal length and distortion) as well as the external parameters between cameras (such as relative position and attitude), thereby enabling 3D reconstruction and visual measurement between cameras.

[0004] General-purpose algorithms can only perform single-camera calibration, which is time-consuming and hinders the iterative upgrade of perception algorithms, failing to meet the calibration requirements of mass-produced vehicles. How to calibrate multiple cameras to improve camera calibration efficiency remains a question that needs to be addressed. Summary of the Invention

[0005] This application provides a multi-camera joint calibration method, apparatus, electronic device, and storage medium to address the problem that general algorithms can only perform single-camera calibration, which is time-consuming, hinders the iterative upgrade of perception algorithms, and fails to meet the calibration requirements of mass-produced vehicles. It addresses the issue of how to calibrate multiple cameras to improve camera calibration efficiency.

[0006] On one hand, this application provides a multi-camera joint calibration method applied to a calibration site, wherein multiple cameras, multiple intrinsic parameter calibration plates, and multiple extrinsic parameter calibration plates are fixedly installed at the calibration site. The different cameras are mounted at different positions on the vehicle, and the different calibration plates are positioned at different locations on the ground. The method includes:

[0007] The camera is calibrated based on multiple intrinsic parameter calibration board images acquired by the camera to obtain the intrinsic parameter matrix of the camera; and the camera is calibrated based on multiple extrinsic parameter calibration board images acquired by the camera and the intrinsic parameter matrix to obtain the extrinsic parameter matrix of the camera, thus completing the calibration of one camera; wherein, both the intrinsic parameter calibration board images and the extrinsic parameter calibration board images are checkerboard images;

[0008] The calibration steps for one camera are repeatedly executed multiple times within the same time period to obtain the intrinsic and extrinsic parameter matrices of each of the multiple cameras, thereby completing the calibration of the multiple cameras.

[0009] In one embodiment, calibrating the camera based on multiple intrinsic parameter calibration board images acquired by the camera to obtain the camera's intrinsic parameter matrix includes:

[0010] Acquire multiple intrinsic parameter calibration plate images captured by the camera;

[0011] Multiple first corner points are identified in each of the intrinsic parameter calibration plate images;

[0012] The intrinsic parameter matrix is ​​determined based on multiple first corner points in each of the intrinsic parameter calibration board images;

[0013] The first corner point is a common vertex of the four smallest quadrilaterals in the intrinsic parameter calibration board image.

[0014] In one embodiment, determining the intrinsic parameter matrix based on a plurality of first corner points in each of the intrinsic parameter calibration board images includes:

[0015] Obtain the scaling factor;

[0016] For a given intrinsic parameter calibration board image, obtain the first coordinates of each first corner point in the intrinsic parameter calibration board image in the pixel coordinate system; and obtain the second coordinates of each first corner point in the intrinsic parameter calibration board image in the world coordinate system; repeat this step until the first and second coordinates of each first corner point in each intrinsic parameter calibration board image are obtained;

[0017] Based on the scale factor, the first coordinate and the second coordinate of each first corner point, the value of each element in the homography matrix is ​​determined. The homography matrix is ​​the matrix obtained by performing matrix multiplication on the intrinsic parameter matrix and the extrinsic parameter matrix.

[0018] The intrinsic parameter matrix is ​​determined based on the value of each element in the homography matrix.

[0019] In one embodiment, determining the intrinsic parameter matrix based on the value of each element in the homography matrix includes:

[0020] A new matrix is ​​constructed by performing matrix multiplication on the transpose of the intrinsic parameter matrix and the inverse of the intrinsic parameter matrix.

[0021] Obtain the matrix relationship between the new matrix and the homography matrix;

[0022] Based on the matrix relationship and the value of each element in the homography matrix, determine the value of each element in the new matrix;

[0023] Based on the value of each element in the new matrix and the matrix relationship between the new matrix and the intrinsic parameter matrix, the value of each element in the intrinsic parameter matrix is ​​determined, and the intrinsic parameter matrix is ​​obtained.

[0024] In one embodiment, determining the value of each element in the homography matrix based on the scale factor, the first coordinates and the second coordinates of each first corner point includes:

[0025] For each of the first corner points, the value of each element in the homography matrix is ​​determined based on the scale factor, the first coordinate and the second coordinate of the first corner point, to obtain an initial homography matrix;

[0026] Repeat the step described above, based on the scale factor, until an initial homography matrix corresponding to each of the first corner points is obtained;

[0027] The steps for determining the value of an element in the homography matrix are as follows: determine the value of an element in the homography matrix based on the value of the element in each initial homography matrix;

[0028] Repeat the steps of determining the value of an element in the homography matrix until the value of each element in the homography matrix is ​​determined.

[0029] In one embodiment, identifying multiple first corner points in each of the intrinsic parameter calibration board images includes:

[0030] The step of identifying multiple first corner points in an intrinsic parameter calibration board image includes: converting the intrinsic parameter calibration board image into a grayscale image, and binarizing the grayscale image into a binary image; processing the binary image based on an edge detection algorithm to obtain a checkerboard outline, and obtaining the outline of each individual checkerboard grid based on an image dilation algorithm; when the shape of the outline is determined to be a quadrilateral based on a polygon approximation algorithm, obtaining a common vertex of every four quadrilaterals as a first corner point, until multiple first corner points are obtained;

[0031] Repeat the step of identifying multiple first corner points in one of the intrinsic parameter calibration board images until multiple first corner points are identified in each of the intrinsic parameter calibration board images.

[0032] In one embodiment, calibrating the camera based on multiple extrinsic calibration board images acquired by the camera and the intrinsic parameter matrix to obtain the camera's extrinsic parameter matrix includes:

[0033] Acquire multiple images of the extrinsic calibration plate captured by the camera;

[0034] Multiple second corner points were identified in each of the extrinsic parameter calibration plate images;

[0035] The extrinsic parameter matrix is ​​determined based on multiple second corner points in each extrinsic parameter calibration board image and the intrinsic parameter matrix;

[0036] The second corner point is a common vertex of the four smallest quadrilaterals in the extrinsic calibration plate image.

[0037] In one embodiment, determining the extrinsic parameter matrix based on a plurality of second corner points in each of the extrinsic parameter calibration board images and the intrinsic parameter matrix includes:

[0038] For a given extrinsic calibration board image, obtain the third coordinates of each second corner point in the extrinsic calibration board image in the pixel coordinate system;

[0039] Based on the intrinsic parameter matrix and the third coordinates of each second corner point, determine the fourth coordinates of each second corner point in the world coordinate system;

[0040] Based on the fourth coordinate of each second corner point, a distance value function is constructed between every two adjacent second corner points, and a linear error function is constructed between each second corner point and the corner points in its row that are collinear; wherein, the collinearity of the corner points is based on a line fitted by multiple second corner points in the same row;

[0041] Using the distance value function and the linear error function as objective functions, optimize the fourth coordinate of each second corner point, and determine the value of each element in the extrinsic parameter matrix based on the optimization results and the value of each element in the intrinsic parameter matrix to obtain an initial extrinsic parameter matrix;

[0042] Repeat the step of obtaining an initial extrinsic matrix for each of the extrinsic calibration board images until an initial extrinsic matrix is ​​obtained for each of the extrinsic calibration board images, and determine the extrinsic matrix based on each of the initial extrinsic matrixes.

[0043] In one embodiment, optimizing the fourth coordinates of each second corner point using the distance value function and the linear error function as objective functions includes:

[0044] The distance value function is repeatedly optimized until the distance difference between any two adjacent second corner points is minimized, at which point the optimization is complete and the initial optimized fourth coordinates of each second corner point are obtained.

[0045] Substitute the initial optimized fourth coordinates of each second corner point into the linear error function, and repeat the optimization of the linear error function. When the distance difference between each second corner point and the corner point of its row is minimized, the optimization is completed, and the target optimized fourth coordinates of each second corner point are obtained as the optimization result.

[0046] In one embodiment, determining the extrinsic matrix based on each of the initial extrinsic matrices includes:

[0047] Obtain the transformation matrix between the extrinsic parameter plate calibration image and the vehicle coordinate system corresponding to each of the initial extrinsic parameter matrices;

[0048] The extrinsic matrix is ​​determined based on each of the initial extrinsic matrixes and the transformation matrix corresponding to each of the initial extrinsic matrixes.

[0049] On the other hand, this application provides a multi-camera joint calibration device applied to a calibration site. The calibration site is fixedly equipped with multiple cameras, multiple intrinsic parameter calibration plates, and multiple extrinsic parameter calibration plates. The different cameras are mounted in different positions on the vehicle, and the different calibration plates are positioned in different positions on the ground. The device includes:

[0050] A single-camera calibration module is used to calibrate the camera based on multiple intrinsic parameter calibration board images acquired by the camera, and obtain the intrinsic parameter matrix of the camera; and to calibrate the camera based on multiple extrinsic parameter calibration board images acquired by the camera and the intrinsic parameter matrix, and obtain the extrinsic parameter matrix of the camera, thus completing the calibration of one camera; wherein, both the intrinsic parameter calibration board images and the extrinsic parameter calibration board images are checkerboard images;

[0051] The multi-camera calibration module repeatedly executes the above calibration steps for one camera multiple times within the same time period to obtain the intrinsic parameter matrix and extrinsic parameter matrix of each of the multiple cameras, thereby completing the calibration of the multiple cameras.

[0052] On the other hand, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0053] The memory stores computer-executed instructions;

[0054] The processor executes computer execution instructions stored in the memory to implement the multi-camera joint calibration method as described in the first aspect.

[0055] On the other hand, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed, cause a computer to perform the multi-camera joint calibration method as described in the first aspect.

[0056] On the other hand, this application provides a computer program product, including a computer program that, when executed by a processor, implements the multi-camera joint calibration method as described in the first aspect.

[0057] In summary, embodiments of this application provide a multi-camera joint calibration method. A calibration site is fixedly equipped with multiple cameras, multiple intrinsic parameter calibration boards, and multiple extrinsic parameter calibration boards. Different cameras are mounted at different positions on the vehicle, and different calibration boards are positioned at different locations on the ground. The multi-camera joint calibration method includes: calibrating the camera based on images acquired by the camera from multiple intrinsic parameter calibration boards to obtain the camera's intrinsic parameter matrix; and calibrating the camera based on images acquired by the camera from multiple extrinsic parameter calibration boards and the intrinsic parameter matrix to obtain the camera's extrinsic parameter matrix, thus completing the calibration of one camera. The calibration steps for one camera are then repeatedly executed multiple times within the same time period to obtain the intrinsic and extrinsic parameter matrices for each of the multiple cameras, thus completing the calibration of multiple cameras.

[0058] In this way, calibrating multiple cameras at the same time reduces the time spent on multi-camera calibration, which is more conducive to the iterative upgrading of perception algorithms and meets the calibration requirements of mass-produced vehicles. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0060] Figure 1 A schematic diagram illustrating an application scenario of the multi-camera joint calibration method provided in this application;

[0061] Figure 2 A flowchart illustrating a multi-camera joint calibration method provided in one embodiment of this application;

[0062] Figure 3 A schematic diagram of the calibration site in a multi-camera joint calibration method provided in one embodiment of this application;

[0063] Figure 4 A flowchart illustrating a multi-camera joint calibration method provided in another embodiment of this application;

[0064] Figure 5 A schematic diagram of an intrinsic parameter calibration plate image in a multi-camera joint calibration method provided in one embodiment of this application;

[0065] Figure 6 A flowchart illustrating a multi-camera joint calibration method provided in yet another embodiment of this application;

[0066] Figure 7A schematic diagram of a multi-camera joint calibration device provided in one embodiment of this application.

[0067] Figure 8 A schematic diagram of an electronic device provided for one embodiment of this application.

[0068] The accompanying drawings have illustrated specific embodiments of this disclosure, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this disclosure to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0069] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0070] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0071] First, let's explain the terms used in this application:

[0072] World coordinate system: A user-defined 3D world coordinate system that describes the position of objects and cameras in the real world. The origin can be chosen arbitrarily.

[0073] Camera coordinate system: The origin is the optical center of the camera. The X and Y axes are parallel to the X and Y axes of the image coordinate system, and the Z axis is the optical axis of the camera.

[0074] Pixel coordinate system: Coincides with the imaging plane, with the upper left corner of the imaging plane as the origin, and the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system, respectively.

[0075] Sensors are the primary means by which intelligent driving systems acquire environmental and target information. Sensors include millimeter-wave radar, lidar, cameras, and inertial measurement units (IMUs). To ensure the accuracy and reliability of the data acquired by sensors, calibration is necessary. Taking cameras as an example, calibration determines the internal parameters of the cameras (such as focal length and distortion) as well as the external parameters between cameras (such as relative position and attitude), thereby enabling 3D reconstruction and visual measurement between cameras. General-purpose algorithms can only perform single-camera calibration, which is time-consuming, hinders the iterative upgrade of perception algorithms, and does not meet the calibration requirements of mass-produced vehicles.

[0076] Based on this, this application provides a multi-camera joint calibration method, apparatus, electronic device, and storage medium. This multi-camera joint calibration method can calibrate multiple cameras simultaneously, reducing the time required for multi-camera calibration, facilitating iterative upgrades of perception algorithms, and meeting the calibration requirements of mass-produced vehicles.

[0077] The multi-camera joint calibration method provided in this application is applied to electronic devices, such as processors installed on logistics vehicles, or cloud servers. Figure 1 This diagram illustrates the application of the multi-camera joint calibration method provided in this application. The calibration site is fixedly equipped with multiple cameras (not shown), multiple intrinsic parameter calibration plates (empty boxes shown), and multiple extrinsic parameter calibration plates (filled black boxes shown). The cameras are mounted at different positions on the vehicle, and the calibration plates are positioned at different locations on the ground. When calibrating one camera, the electronic device calibrates the camera based on images from multiple intrinsic parameter calibration plates acquired by the camera, obtaining the camera's intrinsic parameter matrix; and it further calibrates the camera based on images from multiple extrinsic parameter calibration plates acquired by the camera and the intrinsic parameter matrix, obtaining the camera's extrinsic parameter matrix, thus completing the calibration of one camera. The calibration steps for one camera are then repeatedly executed multiple times within the same time period to obtain the intrinsic and extrinsic parameter matrices for each of the multiple cameras, completing the calibration of multiple cameras.

[0078] Please see Figure 2 One embodiment of this application provides a multi-camera joint calibration method.

[0079] This multi-camera joint calibration method is applied to a calibration site. The calibration site is fixedly equipped with multiple cameras, multiple internal parameter calibration plates, and multiple external parameter calibration plates. The different cameras are mounted in different positions on the vehicle, and the different calibration plates are positioned differently on the ground. For example... Figure 3 The diagram shows a calibration site where a total of seven cameras are installed on the vehicle. Three cameras are installed in the front of the vehicle and their specific locations are different. The other four cameras are installed on the side and rear of the vehicle.

[0080] The calibration site also includes multiple internal parameter calibration boards (such as...). Figure 3 The blank box without fill shown) and multiple extrinsic calibration plates (such as Figure 3 (The black box shown is filled in). The intrinsic calibration board should be set up so that each camera can acquire an image from at least one intrinsic calibration board. The extrinsic calibration board should be set up so that each camera can acquire an image from at least one extrinsic calibration board.

[0081] The multi-camera joint calibration method includes:

[0082] S210, The camera is calibrated based on multiple intrinsic parameter calibration board images acquired by the camera to obtain the intrinsic parameter matrix of the camera, wherein the intrinsic parameter calibration board image is a checkerboard image.

[0083] To improve calibration accuracy, when acquiring multiple images of the intrinsic parameter calibration plate from the camera, at least 10 different images of the intrinsic parameter calibration plate should be obtained. Optionally, one intrinsic parameter calibration plate can be rotated at different angles to allow the camera to acquire multiple different images of the intrinsic parameter calibration plate. Optionally, the angle of the intrinsic parameter calibration plate remains constant, and multiple different intrinsic parameter calibration plates are set to allow the camera to acquire multiple different images of the intrinsic parameter calibration plate. Optionally, there can be multiple intrinsic parameter calibration plates, and each intrinsic parameter calibration plate can be rotated at different angles to allow the camera to acquire multiple different images of the intrinsic parameter calibration plate.

[0084] In an optional embodiment, the three cameras positioned in front of the vehicle respectively acquire close-range, mid-range, and long-range images of the internal parameter calibration plate. The camera positioned on the side of the vehicle acquires close-range images of the internal parameter calibration plate. The camera positioned at the rear of the vehicle acquires mid-range images of the internal parameter calibration plate.

[0085] Obtaining the intrinsic parameter matrix of the camera essentially means obtaining the value of each element in that matrix. Assume the intrinsic parameter matrix is... The purpose of this embodiment is to obtain each element (f) in matrix A based on multiple images of the intrinsic parameter calibration plate. x f y The values ​​of , γ, u0, v0).

[0086] Optionally, multiple points are determined from the intrinsic parameter calibration board image, and the value of each element in the intrinsic parameter matrix is ​​determined based on the two-dimensional coordinates of each point in the pixel coordinate system and the three-dimensional coordinates in the world coordinate system. Optionally, the multiple points can be multiple corner points, which are a common vertex of four smallest quadrilaterals in the intrinsic parameter calibration board image.

[0087] S220, the camera is calibrated based on multiple extrinsic calibration board images and the intrinsic parameter matrix acquired by the camera, and the extrinsic parameter matrix of the camera is obtained, thus completing the calibration of the camera. The extrinsic calibration board image is a checkerboard image.

[0088] To improve calibration accuracy, when acquiring multiple images of the extrinsic calibration plate from the camera, at least 10 different images of the extrinsic calibration plate should be obtained. Optionally, one extrinsic calibration plate can be rotated at different angles to allow the camera to acquire multiple different images of the extrinsic calibration plate. Optionally, the angle of the extrinsic calibration plate remains constant, and multiple different extrinsic calibration plates are set to allow the camera to acquire multiple different images of the extrinsic calibration plate. Optionally, there can be multiple extrinsic calibration plates, and each extrinsic calibration plate can be rotated at different angles to allow the camera to acquire multiple different images of the extrinsic calibration plate.

[0089] The extrinsic parameter matrix is ​​R = [r1, r2, r3]. Determining this extrinsic parameter matrix essentially means determining the value of each element (r1, r2, r3) in the extrinsic parameter matrix. After obtaining the intrinsic parameter matrix, the value of each element in the extrinsic parameter matrix can be determined based on the intrinsic parameter matrix and the extrinsic parameter calibration plate image.

[0090] Optionally, multiple points are determined from the extrinsic parameter calibration board image, and the value of each element in the extrinsic parameter matrix is ​​determined based on the two-dimensional coordinates of each point in the pixel coordinate system and the three-dimensional coordinates in the world coordinate system. Optionally, the multiple points can be multiple corner points, which are a common vertex of four smallest quadrilaterals in the extrinsic parameter calibration board image.

[0091] S230 repeatedly executes the calibration steps for a camera multiple times within the same time period to obtain the intrinsic and extrinsic parameter matrices of each of the multiple cameras, thus completing the calibration of multiple cameras.

[0092] Calibrate multiple cameras, meaning calibrate each camera within the same time period. The camera calibration steps provided in this embodiment include obtaining the camera's intrinsic parameter matrix and obtaining the camera's extrinsic parameter matrix.

[0093] Optionally, each camera can be calibrated simultaneously to reduce the time required for multi-camera calibration. Alternatively, multiple cameras can be calibrated within a specified time period, which can also reduce the time required for multi-camera calibration.

[0094] In summary, this embodiment provides a multi-camera joint calibration method. The calibration site is fixedly equipped with multiple cameras, multiple intrinsic parameter calibration boards, and multiple extrinsic parameter calibration boards. Different cameras are mounted at different positions on the vehicle, and different calibration boards are positioned at different locations on the ground. The multi-camera joint calibration method includes: calibrating the camera based on images acquired by the camera from multiple intrinsic parameter calibration boards to obtain the camera's intrinsic parameter matrix; and calibrating the camera based on images acquired by the camera from multiple extrinsic parameter calibration boards and the intrinsic parameter matrix to obtain the camera's extrinsic parameter matrix, thus completing the calibration of one camera. The calibration steps for one camera are then repeatedly executed multiple times within the same time period to obtain the intrinsic and extrinsic parameter matrices for each of the multiple cameras, thus completing the calibration of multiple cameras.

[0095] In this way, calibrating multiple cameras at the same time reduces the time spent on multi-camera calibration, which is more conducive to the iterative upgrading of perception algorithms and meets the calibration requirements of mass-produced vehicles.

[0096] Please see Figure 4 An embodiment of this application also provides a multi-camera joint calibration method, further describing the steps of obtaining the intrinsic parameter matrix of the camera.

[0097] Specifically, the steps to obtain the intrinsic parameter matrix of the camera include:

[0098] S410: Acquire multiple intrinsic parameter calibration plate images captured by the camera.

[0099] As described in the above embodiments, in order to improve the accuracy of calibration, when acquiring multiple images of the intrinsic parameter calibration plate captured by the camera, at least 10 different images of the intrinsic parameter calibration plate need to be acquired.

[0100] S420 identifies multiple first corner points in each of the intrinsic parameter calibration plate images.

[0101] like Figure 5 As shown, the first corner point is a common vertex of the four smallest quadrilaterals in the intrinsic parameter calibration board image. That is, as long as four smallest quadrilaterals have a common vertex, then that common vertex is a first corner point.

[0102] When identifying the first corner point in the image of the intrinsic parameter calibration plate, it is necessary to first identify four adjacent quadrilaterals (such as...). Figure 5 (As shown in region A), then identify the common vertex of the four adjacent quadrilaterals.

[0103] In an optional embodiment, the step of identifying multiple first corner points in an intrinsic parameter calibration plate image includes: converting the intrinsic parameter calibration plate image into a grayscale image and binarizing the grayscale image into a binary image; processing the binary image based on an edge detection algorithm to obtain a checkerboard outline, and obtaining the outline of each individual checkerboard grid based on an image dilation algorithm; when the shape of the outline is determined to be a quadrilateral based on a polygon approximation algorithm, obtaining a common vertex of every four quadrilaterals as a first corner point, until multiple first corner points are obtained.

[0104] Specifically, firstly, when converting the intrinsic parameter calibration board image to a grayscale image, the image is input into the software, which then converts it to grayscale. The software averages the pixel values ​​of the RGB channels of the intrinsic parameter calibration board image to obtain the grayscale image.

[0105] Binarizing the grayscale image means converting it into a binary image. Specifically, pixels with grayscale values ​​greater than a preset threshold are set to maximum grayscale values, and pixels with grayscale values ​​less than the preset threshold are set to minimum grayscale values, thus achieving binarization.

[0106] The binary image is then processed using the Canny edge detection algorithm to obtain the checkerboard outline. This binary image includes regions with a grayscale value of 0 and regions with a grayscale value of 255.

[0107] Based on the image dilation algorithm, the contour of each individual chessboard square is obtained. That is, the image dilation algorithm (image dilation operation) is applied to the regions with a gray value of 0 in the chessboard image to separate the connected chessboard contours and obtain the contour of each individual chessboard square.

[0108] After obtaining the outline of a single chessboard square, a polygon approximation algorithm is used to approximate the outline into an approximate polygon. If the number of sides is 4, it is confirmed to be a quadrilateral; otherwise, it is ignored. When the polygon approximation algorithm determines that the shape of the outline is a quadrilateral, adjacent quadrilaterals are found, and the number of adjacent quadrilaterals is recorded. When four mutually adjacent quadrilaterals are found, a common vertex of the four mutually adjacent quadrilaterals is determined as the first corner point.

[0109] Multiple first corner points are identified in each of the intrinsic parameter calibration board images. That is, the step of identifying multiple first corner points in one of the intrinsic parameter calibration board images is repeatedly performed until multiple first corner points in each of the intrinsic parameter calibration board images are identified. Optionally, each of the first corner points in each of the intrinsic parameter calibration board images can be identified, or a preset number of first corner points in each of the intrinsic parameter calibration board images can be identified.

[0110] S430, determine the intrinsic parameter matrix based on multiple first corner points in each intrinsic parameter calibration plate image.

[0111] First, obtain the two-dimensional coordinates (u, v) of each of the multiple first corner points in the pixel coordinate system, and the three-dimensional coordinates (x, v) of each first corner point in the world coordinate system. w ,y w ,0). Then, based on the two-dimensional coordinates (u,v) and three-dimensional coordinates (x,v) of each of the multiple first corner points. w ,y w ,0) determines the value of each element (α, β, γ, u0, v0) in the intrinsic parameter matrix A.

[0112] The origin of the world coordinate system can be any of the first corner points. Optionally, the origin of the world coordinate system provided in this embodiment can be the first corner point located in the first column and first row of the chessboard image.

[0113] In an optional embodiment, the scale factor is first obtained. Then, for an intrinsic parameter calibration board image, the first coordinates of each first corner point in the intrinsic parameter calibration board image in the pixel coordinate system are obtained, and the second coordinates of each first corner point in the intrinsic parameter calibration board image in the world coordinate system are obtained. This step is repeated until the first and second coordinates of each first corner point in each intrinsic parameter calibration board image are obtained. The first coordinates of the first corner point can be represented as (u, v), and the second coordinates of the first corner point can be represented as (x, v). w ,y w ,0).

[0114] Next, based on the scale factor and the first and second coordinates of each first corner point, the value of each element in the homography matrix is ​​determined. This homography matrix is ​​obtained by multiplying the intrinsic parameter matrix and the extrinsic parameter matrix. Finally, based on the value of each element in the homography matrix, the intrinsic parameter matrix is ​​determined.

[0115] Assume the scale factor is s, the homography matrix is ​​H, the intrinsic parameter matrix is ​​A, and the initial extrinsic parameter matrix is ​​R. The initial extrinsic parameter matrix R represents the extrinsic parameter matrix of the calibration image (whether it is the intrinsic parameter calibration plate image or the extrinsic parameter calibration image).

[0116] Wherein, equation (1) is Equation (2) is

[0117] The scaling factor s is known. That is, the homogeneous coordinates of pixel coordinates (u,v), which, when substituted into the first coordinates (u,v) of the first corner point, can determine... Substitute the second coordinate (x) of the first corner pointw ,y w After ,0), it can be determined that Given the known s, and Substituting into equation (2), we can determine The value of each element in the data, and then based on... Determine the intrinsic parameter matrix A.

[0118] The steps for determining the value of each element in the homography matrix H include: for each first corner point, determining the value of each element in the homography matrix based on the scale factor, the first coordinate, and the second coordinate of the first corner point, to obtain an initial homography matrix; repeating the step of determining the value of each element based on the scale factor until an initial homography matrix corresponding to each first corner point is obtained; the steps for determining the value of an element in the homography matrix include: determining the value of an element in the homography matrix based on the value of an element in each initial homography matrix; repeating the steps for determining the value of an element in the homography matrix until the value of each element in the homography matrix is ​​determined.

[0119] That is, the first and second coordinates of each first corner point in each intrinsic parameter calibration plate image need to be substituted to determine the value of each element in the homography matrix. Specifically, by substituting the first and second coordinates of each first corner point in each intrinsic parameter calibration plate image into equation (2), multiple values ​​can be determined for each element in the homography matrix. Optionally, the average of the multiple values ​​determined for an element can be taken as the value of that element. For example, h 11 If there are M values, then h 11 The value is the average of the M values.

[0120] More specifically, determining the intrinsic parameter matrix based on the value of each element in the homography matrix includes: performing matrix multiplication on the transpose and inverse of the intrinsic parameter matrix to construct a new matrix; obtaining the matrix relationship between the new matrix and the homography matrix; determining the value of each element in the new matrix based on the matrix relationship and the value of each element in the homography matrix; and determining the value of each element in the intrinsic parameter matrix based on the value of each element in the new matrix and the matrix relationship between the new matrix and the intrinsic parameter matrix, thus obtaining the intrinsic parameter matrix.

[0121] Assuming the new matrix is ​​B, equation (3) Formula (4) can be derived through reasoning. Where, b = [B 11 B 12 B 22 B 13 B 23 B 33 ] T,

[0122] Seeking, γ=-B 12 f x 2 f y / λ, u0=γv0 / β-B 13 f x 2 / λ,λ=B 33 -[B 13 2 +v0(B 12 B 13 -B 11 B 23 )] / B 11 .

[0123] Since it has been determined The value of each element can be determined using the formula. Determine v 11 and v 12 The value of b is then derived from equation (4). Based on b = [B], the value of b can be deduced. 11 B 12 B 22 B 13 B 23 B 33 ] T Element B can be determined 11 B 12 B 22 B 13 B 23 B 33 The value of .

[0124] This intrinsic parameter matrix

[0125] according to Known B 11 B 12 B 13 B 22 B 23 This allows us to determine the value of element v0 in the intrinsic parameter matrix A.

[0126] According to λ=B 33 -[B 13 2 +v0(B 12 B 13 -B 11 B 23 )] / B 11The value of λ can be determined.

[0127] according to Known λ, known B 11 B 12 B 13 B 23 B 33 This allows us to determine the element f in the intrinsic parameter matrix A. x .

[0128] according to Known λ, known B 11 B 12 B 22 This allows us to determine the element f in the intrinsic parameter matrix A. y .

[0129] According to γ=-B 12 f x 2 f y / λ, known λ, known B 12 f x and f y This allows us to determine the element γ in the intrinsic parameter matrix A.

[0130] In summary, this embodiment provides a method for obtaining the camera's intrinsic parameter matrix in the multi-camera joint calibration method. The value of each element in the intrinsic parameter matrix is ​​determined based on the first and second coordinates of multiple first corner points in each intrinsic parameter calibration image. Compared to traditional methods for determining the camera's intrinsic parameter matrix, the method provided in this embodiment for obtaining the camera's intrinsic parameter matrix is ​​more accurate.

[0131] Please see Figure 6 One embodiment of this application provides a multi-camera joint calibration method, and further describes the steps of how to obtain the camera's extrinsic parameter matrix.

[0132] Specifically, the steps to obtain the extrinsic parameter matrix of the camera include:

[0133] S610 acquires multiple images of the external parameter calibration plate captured by the camera.

[0134] As described in the above embodiment, to ensure the accuracy of the determined extrinsic parameter matrix, the camera needs to capture at least 10 frames of images of the extrinsic parameter calibration board.

[0135] S620 identifies multiple second corner points in each of the extrinsic calibration plate images.

[0136] The second corner point is a common vertex of the four smallest quadrilaterals in the extrinsic calibration plate image. The method for identifying the second corner point in the extrinsic calibration plate image is similar to the method for identifying the first corner point in the intrinsic calibration plate image.

[0137] In an optional embodiment, the step of identifying multiple second corner points in an extrinsic calibration plate image includes: converting the extrinsic calibration plate image into a grayscale image and binarizing the grayscale image into a binary image; processing the binary image based on an edge detection algorithm to obtain a checkerboard outline, and obtaining the outline of each individual checkerboard grid based on an image dilation algorithm; when the shape of the outline is determined to be a quadrilateral based on a polygon approximation algorithm, obtaining a common vertex of every four quadrilaterals as a second corner point, until multiple second corner points are obtained.

[0138] Specifically, firstly, when converting the extrinsic calibration board image to a grayscale image, the image is input into the software, which then converts it to grayscale. The software averages the pixel values ​​of the RGB channels of the extrinsic calibration board image to obtain the grayscale image.

[0139] Binarizing the grayscale image means converting it into a binary image. Specifically, pixels with grayscale values ​​greater than a preset threshold are set to maximum grayscale values, and pixels with grayscale values ​​less than the preset threshold are set to minimum grayscale values, thus achieving binarization.

[0140] The binary image is then processed using the Canny edge detection algorithm to obtain the checkerboard outline. This binary image includes regions with a grayscale value of 0 and regions with a grayscale value of 255.

[0141] Based on the image dilation algorithm, the contour of each individual chessboard square is obtained. That is, the image dilation algorithm (image dilation operation) is applied to the regions with a gray value of 0 in the chessboard image to separate the connected chessboard contours and obtain the contour of each individual chessboard square.

[0142] After obtaining the outline of a single chessboard square, a polygon approximation algorithm is used to approximate the outline into an approximate polygon. If the number of sides is 4, it is confirmed to be a quadrilateral; otherwise, it is ignored. When the polygon approximation algorithm determines that the shape of the outline is a quadrilateral, adjacent quadrilaterals are found, and the number of adjacent quadrilaterals is recorded. When four mutually adjacent quadrilaterals are found, a common vertex of the four mutually adjacent quadrilaterals is determined as the second corner point.

[0143] Multiple second corner points are identified in each of the extrinsic calibration plate images. That is, the step of identifying multiple second corner points in one of the extrinsic calibration plate images is repeated until multiple second corner points in each of the extrinsic calibration plate images are identified. Optionally, each of the second corner points in each of the extrinsic calibration plate images can be identified, or a preset number of the second corner points in each of the extrinsic calibration plate images can be identified.

[0144] S630, the extrinsic parameter matrix is ​​determined based on the multiple second corner points and the intrinsic parameter matrix in each extrinsic parameter calibration plate image.

[0145] For a given extrinsic calibration board image, obtain the third coordinates of each second corner point in the pixel coordinate system. Based on the intrinsic parameter matrix and the third coordinates of each second corner point, determine the fourth coordinates of each second corner point in the world coordinate system. Based on the fourth coordinates of each second corner point, construct a distance function between every two adjacent second corner points, and construct a linear error function between each second corner point and the corner points in its row that are collinear. Here, the collinearity of corner points is based on a line fitted from multiple second corner points in the same row.

[0146] This distance value function

[0147] Where (l) represents the horizontal position of the second corner point, (w) represents the vertical position of the second corner point, (n) represents the number of the second corner points in each row (horizontal) of the extrinsic parameter calibration board image, and (m) represents the number of the second corner points in each column (vertical) of the extrinsic parameter calibration board image. (l, w) can locate a corner point. For example, if n = 9, m = 6, l = 2, and w = 3, then (l, w) represents the second corner point located in the 2nd row and 3rd column.

[0148] d represents the distance difference between any two adjacent corner points in the world coordinate system, determined by the fourth coordinates of those two corner points. D represents the actual measured distance difference between those two corner points. The units of d and D are the same.

[0149] The linear error function

[0150] Among them, L r It is the equation of the line fitted based on the second corner point in the same line, where Q is the fourth coordinate of the second corner point and D represents the concept of distance difference.

[0151] Using the distance value function and the linear error function as objective functions, optimize the fourth coordinate of each second corner point.

[0152] When optimizing the fourth coordinate of each second corner point using the distance value function as the objective function, the distance value function is repeatedly optimized. When the distance difference between any two adjacent second corner points is minimized, the optimization is completed, and the initial optimized fourth coordinate of each second corner point is obtained.

[0153] Then, substitute the initial optimized fourth coordinate of each second corner point into the linear error function, and repeat the optimization of the linear error function. When the distance difference between each second corner point and the corner point of the row is minimized, the optimization is completed, and the target optimized fourth coordinate of each second corner point is obtained as the optimization result.

[0154] Based on the optimization result and the value of each element in the intrinsic parameter matrix, the value of each element in the extrinsic parameter matrix is ​​determined, resulting in an initial extrinsic parameter matrix. Specifically, this is based on the homography matrix described in the above embodiment. The known intrinsic parameter matrix A, the known The initial intrinsic parameter matrix R can be determined. The initial extrinsic parameter matrix R represents the extrinsic parameter matrix of the calibration image (whether it is the intrinsic parameter calibration board image or the extrinsic parameter calibration image), and here it represents the extrinsic parameter matrix of the extrinsic parameter calibration board.

[0155] Finally, the extrinsic parameter matrix is ​​determined based on each initial extrinsic parameter matrix. That is, the transformation matrix between the calibration image on the extrinsic parameter board and the vehicle coordinate system corresponding to each initial extrinsic parameter matrix is ​​obtained. Optionally, a total station can be used to measure the coordinates of the ground checkerboard origin (calibration image origin) in the vehicle coordinate system, and the transformation matrix of the checkerboard coordinate system (calibration image coordinate system) relative to the vehicle coordinate system can be determined. Then, based on each initial extrinsic parameter matrix and its corresponding transformation matrix, the extrinsic parameter matrix is ​​determined, ultimately obtaining the extrinsic parameter matrix of the camera.

[0156] In summary, the method for obtaining the extrinsic parameter matrix in the multi-camera joint calibration method provided in this embodiment involves identifying the second corner point and constructing a distance value function and a linear error function based on the fourth coordinates (coordinates in the world coordinate system) of the second corner point. By continuously optimizing the distance value function, the difference between the first distance difference (the distance difference determined by the fourth coordinates of the two second corner points) and the second distance difference (the actual measured distance difference between the two second corner points) between any two calculated second corner points is minimized. Similarly, by continuously optimizing the linear error function, the distance difference between any second corner point and the collinear corner point is minimized. In this way, a more accurate extrinsic parameter matrix for the camera can be obtained.

[0157] Please see Figure 7One embodiment of this application provides a multi-camera joint calibration device 10, which is applied to a calibration site. The calibration site is fixedly equipped with multiple cameras, multiple internal parameter calibration plates and multiple external parameter calibration plates. The different cameras are installed in different positions on the vehicle, and the different calibration plates are set in different positions on the ground.

[0158] The device includes:

[0159] The single-camera calibration module 100 is used to calibrate the camera based on multiple intrinsic parameter calibration board images acquired by the camera, and obtain the intrinsic parameter matrix of the camera; and to calibrate the camera based on multiple extrinsic parameter calibration board images acquired by the camera and the intrinsic parameter matrix, and obtain the extrinsic parameter matrix of the camera, thus completing the calibration of one camera; wherein, both the intrinsic parameter calibration board images and the extrinsic parameter calibration board images are checkerboard images.

[0160] The multi-camera calibration module 200 is used to repeatedly execute the above calibration steps for one camera multiple times within the same time period to obtain the intrinsic parameter matrix and extrinsic parameter matrix of each of the multiple cameras, thereby completing the calibration of multiple cameras.

[0161] Optionally, the single-camera calibration module 100 is specifically used to: acquire multiple intrinsic parameter calibration board images collected by the camera; identify multiple first corner points in each intrinsic parameter calibration board image; determine the intrinsic parameter matrix based on the multiple first corner points in each intrinsic parameter calibration board image; wherein, the first corner point is a common vertex of four smallest quadrilaterals in the intrinsic parameter calibration board image.

[0162] Optionally, the single-camera calibration module 100 is specifically used for: obtaining a scale factor; for an intrinsic calibration board image, obtaining the first coordinates of each first corner point in the intrinsic calibration board image in the pixel coordinate system; and obtaining the second coordinates of each first corner point in the intrinsic calibration board image in the world coordinate system; repeating this step until the first and second coordinates of each first corner point in each intrinsic calibration board image are obtained; determining the value of each element in the homography matrix based on the scale factor, the first coordinates and the second coordinates of each first corner point, wherein the homography matrix is ​​the matrix obtained by matrix multiplication of the intrinsic parameter matrix and the extrinsic parameter matrix; and determining the intrinsic parameter matrix based on the value of each element in the homography matrix.

[0163] Optionally, the single-camera calibration module 100 is specifically used for: performing matrix multiplication on the transpose of the intrinsic parameter matrix and the inverse of the intrinsic parameter matrix to construct a new matrix; obtaining the matrix relationship between the new matrix and the homography matrix; determining the value of each element in the new matrix based on the matrix relationship and the value of each element in the homography matrix; and determining the value of each element in the intrinsic parameter matrix based on the value of each element in the new matrix and the matrix relationship between the new matrix and the intrinsic parameter matrix, thereby obtaining the intrinsic parameter matrix.

[0164] Optionally, the single-camera calibration module 100 is specifically configured to: for each of the first corner points, determine the value of each element in the homography matrix based on the scale factor, the first coordinate, and the second coordinate of the first corner point, and obtain an initial homography matrix; repeat the step of determining the value of an element in the homography matrix based on the scale factor until an initial homography matrix corresponding to each of the first corner points is obtained; for the step of determining the value of an element in the homography matrix: determine the value of an element in the homography matrix based on the value of an element in each of the initial homography matrices; repeat the step of determining the value of an element in the homography matrix until the value of each element in the homography matrix is ​​determined.

[0165] Optionally, the single-camera calibration module 100 is specifically used for: the step of identifying multiple first corner points in an intrinsic parameter calibration board image, including: converting the intrinsic parameter calibration board image into a grayscale image, and binarizing the grayscale image into a binary image; processing the binary image based on an edge detection algorithm to obtain a checkerboard outline, and obtaining the outline of each individual checkerboard grid based on an image dilation algorithm; when the shape of the outline is determined to be a quadrilateral based on a polygon approximation algorithm, obtaining a common vertex of every four quadrilaterals as a first corner point, until multiple first corner points are obtained; repeatedly performing the step of identifying multiple first corner points in an intrinsic parameter calibration board image, until multiple first corner points in each intrinsic parameter calibration board image are identified.

[0166] Optionally, the single-camera calibration module 100 is specifically used to: acquire multiple extrinsic calibration board images captured by the camera; identify multiple second corner points in each extrinsic calibration board image; determine the extrinsic parameter matrix based on the multiple second corner points in each extrinsic calibration board image and the intrinsic parameter matrix; wherein, the second corner point is a common vertex of four smallest quadrilaterals in the extrinsic calibration board image.

[0167] Optionally, the single-camera calibration module 100 is specifically used for: for an extrinsic calibration board image, obtaining the third coordinates of each second corner point in the pixel coordinate system; determining the fourth coordinates of each second corner point in the world coordinate system based on the intrinsic parameter matrix and the third coordinates of each second corner point; constructing a distance value function between every two adjacent second corner points based on the fourth coordinates of each second corner point, and constructing a linear error function between each second corner point and the corner points in its row that are collinear; wherein, the corner point collinearity is based on a line fitted by multiple second corner points in the same row; optimizing the fourth coordinates of each second corner point with the distance value function and the linear error function as objective functions, and determining the value of each element in the extrinsic parameter matrix based on the optimization result and the value of each element in the intrinsic parameter matrix, thereby obtaining an initial extrinsic parameter matrix; repeatedly executing the step of obtaining the initial extrinsic parameter matrix for an extrinsic calibration board image until an initial extrinsic parameter matrix corresponding to each extrinsic calibration board image is obtained, and determining the extrinsic parameter matrix based on each initial extrinsic parameter matrix.

[0168] Optionally, the single-camera calibration module 100 is specifically used to: repeatedly optimize the distance value function, and when the distance difference between any two adjacent second corner points is minimized, complete the optimization and obtain the initial optimized fourth coordinates of each second corner point; substitute the initial optimized fourth coordinates of each second corner point into the linear error function, repeatedly optimize the linear error function, and when the distance difference between each second corner point and the corner point of its row that is collinear is minimized, complete the optimization and obtain the target optimized fourth coordinates of each second corner point as the optimization result.

[0169] Optionally, the single-camera calibration module 100 is specifically used to: obtain the transformation matrix between the extrinsic parameter plate calibration image corresponding to each initial extrinsic parameter matrix and the vehicle coordinate system; and determine the extrinsic parameter matrix based on each initial extrinsic parameter matrix and the transformation matrix corresponding to each initial extrinsic parameter matrix.

[0170] Please see Figure 8 One embodiment of this application also provides an electronic device 20, which includes a processor 21 and a memory 22 communicatively connected to the processor 21. The memory 22 stores computer-executable instructions, and the processor 21 executes the computer-executable instructions stored in the memory 22 to implement the multi-camera joint calibration method provided in any of the above embodiments.

[0171] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed, cause the computer-executable instructions to be executed by a processor to implement the multi-camera joint calibration method provided in any of the preceding embodiments.

[0172] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-camera joint calibration method as provided in any of the preceding embodiments.

[0173] It should be noted that the aforementioned computer-readable storage media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc. It can also be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0174] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0175] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0177] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0179] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0180] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multi-camera joint calibration method, characterized in that, The method is applied to a calibration site, where multiple cameras, multiple internal parameter calibration plates, and multiple external parameter calibration plates are fixedly installed. The cameras are mounted in different positions on the vehicle, and the calibration plates are positioned in different positions on the ground. The method includes: The camera is calibrated based on multiple intrinsic parameter calibration board images acquired by the camera to obtain the intrinsic parameter matrix of the camera; and the camera is calibrated based on multiple extrinsic parameter calibration board images acquired by the camera and the intrinsic parameter matrix to obtain the extrinsic parameter matrix of the camera, thus completing the calibration of one camera; wherein, both the intrinsic parameter calibration board images and the extrinsic parameter calibration board images are checkerboard images; The above calibration steps for one camera are repeatedly executed multiple times within the same time period to obtain the intrinsic parameter matrix and extrinsic parameter matrix of each of the multiple cameras, thereby completing the calibration of multiple cameras; The step of calibrating the camera based on multiple extrinsic calibration board images acquired by the camera and the intrinsic parameter matrix, and obtaining the extrinsic parameter matrix of the camera, includes: Acquire multiple images of the extrinsic calibration plate captured by the camera; Multiple second corner points were identified in each of the extrinsic parameter calibration plate images; The extrinsic parameter matrix is ​​determined based on multiple second corner points in each extrinsic parameter calibration board image and the intrinsic parameter matrix; The second corner point is a common vertex of the four smallest quadrilaterals in the extrinsic calibration plate image; Determining the extrinsic parameter matrix based on multiple second corner points in each extrinsic parameter calibration board image and the intrinsic parameter matrix includes: For a given extrinsic calibration board image, obtain the third coordinates of each second corner point in the extrinsic calibration board image in the pixel coordinate system; Based on the intrinsic parameter matrix and the third coordinates of each second corner point, determine the fourth coordinates of each second corner point in the world coordinate system; Based on the fourth coordinate of each second corner point, a distance value function is constructed between every two adjacent second corner points, and a linear error function is constructed between each second corner point and the corner points in its row that are collinear; wherein, the collinearity of the corner points is based on a line fitted by multiple second corner points in the same row; Using the distance value function and the linear error function as objective functions, optimize the fourth coordinate of each second corner point, and determine the value of each element in the extrinsic parameter matrix based on the optimization results and the value of each element in the intrinsic parameter matrix to obtain an initial extrinsic parameter matrix; Repeat the step of obtaining an initial extrinsic matrix for each of the extrinsic calibration board images until an initial extrinsic matrix is ​​obtained for each of the extrinsic calibration board images, and determine the extrinsic matrix based on each of the initial extrinsic matrixes.

2. The method according to claim 1, characterized in that, The step of calibrating the camera based on multiple intrinsic parameter calibration board images acquired by the camera to obtain the camera's intrinsic parameter matrix includes: Acquire multiple intrinsic parameter calibration plate images captured by the camera; Multiple first corner points are identified in each of the intrinsic parameter calibration plate images; The intrinsic parameter matrix is ​​determined based on multiple first corner points in each of the intrinsic parameter calibration board images; The first corner point is a common vertex of the four smallest quadrilaterals in the intrinsic parameter calibration board image.

3. The method according to claim 2, characterized in that, Determining the intrinsic parameter matrix based on multiple first corner points in each intrinsic parameter calibration board image includes: Obtain the scaling factor; For a given intrinsic parameter calibration board image, obtain the first coordinates of each first corner point in the intrinsic parameter calibration board image in the pixel coordinate system; and obtain the second coordinates of each first corner point in the intrinsic parameter calibration board image in the world coordinate system; repeat this step until the first and second coordinates of each first corner point in each intrinsic parameter calibration board image are obtained; Based on the scale factor, the first coordinate and the second coordinate of each first corner point, the value of each element in the homography matrix is ​​determined. The homography matrix is ​​the matrix obtained by performing matrix multiplication on the intrinsic parameter matrix and the extrinsic parameter matrix. The intrinsic parameter matrix is ​​determined based on the value of each element in the homography matrix.

4. The method according to claim 3, characterized in that, Determining the intrinsic parameter matrix based on the value of each element in the homography matrix includes: A new matrix is ​​constructed by performing matrix multiplication on the transpose of the intrinsic parameter matrix and the inverse of the intrinsic parameter matrix. Obtain the matrix relationship between the new matrix and the homography matrix; Based on the matrix relationship and the value of each element in the homography matrix, determine the value of each element in the new matrix; Based on the value of each element in the new matrix and the matrix relationship between the new matrix and the intrinsic parameter matrix, the value of each element in the intrinsic parameter matrix is ​​determined, and the intrinsic parameter matrix is ​​obtained.

5. The method according to claim 3, characterized in that, The step of determining the value of each element in the homography matrix based on the scale factor, the first coordinate and the second coordinate of each first corner point includes: For each of the first corner points, the value of each element in the homography matrix is ​​determined based on the scale factor, the first coordinate and the second coordinate of the first corner point, to obtain an initial homography matrix; Repeat the step described above, based on the scale factor, until an initial homography matrix corresponding to each of the first corner points is obtained; The steps for determining the value of an element in the homography matrix are as follows: determine the value of an element in the homography matrix based on the value of the element in each initial homography matrix; Repeat the steps of determining the value of an element in the homography matrix until the value of each element in the homography matrix is ​​determined.

6. The method according to claim 2, characterized in that, The identification of multiple first corner points in each of the intrinsic parameter calibration plate images includes: The step of identifying multiple first corner points in an intrinsic parameter calibration board image includes: converting the intrinsic parameter calibration board image into a grayscale image, and binarizing the grayscale image into a binary image; processing the binary image based on an edge detection algorithm to obtain a checkerboard outline, and obtaining the outline of each individual checkerboard grid based on an image dilation algorithm; when the shape of the outline is determined to be a quadrilateral based on a polygon approximation algorithm, obtaining a common vertex of every four quadrilaterals as a first corner point, until multiple first corner points are obtained; Repeat the step of identifying multiple first corner points in one of the intrinsic parameter calibration board images until multiple first corner points are identified in each of the intrinsic parameter calibration board images.

7. The method according to claim 1, characterized in that, The step of optimizing the fourth coordinate of each second corner point using the distance value function and the linear error function as objective functions includes: The distance value function is repeatedly optimized until the distance difference between any two adjacent second corner points is minimized, at which point the optimization is complete and the initial optimized fourth coordinates of each second corner point are obtained. Substitute the initial optimized fourth coordinates of each second corner point into the linear error function, and repeat the optimization of the linear error function. When the distance difference between each second corner point and the corner point of its row is minimized, the optimization is completed, and the target optimized fourth coordinates of each second corner point are obtained as the optimization result.

8. The method according to claim 1, characterized in that, Determining the extrinsic matrix based on each of the initial extrinsic matrices includes: Obtain the transformation matrix between the extrinsic parameter plate calibration image and the vehicle coordinate system corresponding to each of the initial extrinsic parameter matrices; The extrinsic matrix is ​​determined based on each of the initial extrinsic matrixes and the transformation matrix corresponding to each of the initial extrinsic matrixes.

9. A multi-camera joint calibration device, characterized in that, Applied to a calibration site, the calibration site is fixedly equipped with multiple cameras, multiple internal parameter calibration plates, and multiple external parameter calibration plates. The different cameras are mounted in different positions on the vehicle, and the different calibration plates are positioned in different positions on the ground. The device includes: A single-camera calibration module is used to calibrate the camera based on multiple intrinsic parameter calibration board images acquired by the camera, and obtain the intrinsic parameter matrix of the camera; and to calibrate the camera based on multiple extrinsic parameter calibration board images acquired by the camera and the intrinsic parameter matrix, and obtain the extrinsic parameter matrix of the camera, thus completing the calibration of one camera; wherein, both the intrinsic parameter calibration board images and the extrinsic parameter calibration board images are checkerboard images; The multi-camera calibration module repeatedly executes the above calibration steps for one camera multiple times within the same time period to obtain the intrinsic parameter matrix and extrinsic parameter matrix of each of the multiple cameras, thereby completing the calibration of the multiple cameras; The single-camera calibration module is specifically used to acquire multiple extrinsic calibration board images captured by the camera; identify multiple second corner points in each extrinsic calibration board image; and determine the extrinsic parameter matrix based on the multiple second corner points in each extrinsic calibration board image and the intrinsic parameter matrix; wherein, the second corner point is a common vertex of four smallest quadrilaterals in the extrinsic calibration board image; The single-camera calibration module is specifically used to: obtain the third coordinates of each second corner point in the pixel coordinate system for a given extrinsic calibration board image; determine the fourth coordinates of each second corner point in the world coordinate system based on the intrinsic parameter matrix and the third coordinates of each second corner point; construct a distance function between every two adjacent second corner points based on the fourth coordinates of each second corner point, and construct a linear error function between each second corner point and the corner points in its row that are collinear; wherein, the collinearity of the corner points is based on a line fitted by multiple second corner points in the same row; optimize the fourth coordinates of each second corner point using the distance function and the linear error function as objective functions, and determine the value of each element in the extrinsic parameter matrix based on the optimization result and the value of each element in the intrinsic parameter matrix to obtain an initial extrinsic parameter matrix; repeat the step of obtaining the initial extrinsic parameter matrix for a given extrinsic calibration board image until an initial extrinsic parameter matrix is ​​obtained for each extrinsic calibration board image, and determine the extrinsic parameter matrix based on each initial extrinsic parameter matrix.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the multi-camera joint calibration method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed, cause the computer to perform the multi-camera joint calibration method as described in any one of claims 1-8.

Images