A camera calibration method, device, equipment and medium

Abstract

An embodiment of the present application provides a camera calibration method, device, equipment and medium, which relates to the field of computer technology, and includes: obtaining the conversion relationship between the reference camera coordinate system of the reference camera and the calibration board coordinate system of each calibration board, and calculating the first conversion relationship between the calibration board coordinate systems of at least two calibration boards; determining the reference conversion relationship between the calibration board coordinate systems of each calibration board and the reference calibration board coordinate system of the reference calibration board, and calculating the reference coordinates of each calibration point in each calibration board in the reference calibration board coordinate system; using the target image and the reference coordinates of each calibration point in the target calibration board to obtain the second conversion relationship between the camera coordinate system of the to-be-calibrated camera and the reference calibration board coordinate system; calculating the target conversion relationship between the camera coordinate systems of the two to-be-calibrated cameras. Applying the solution provided by the embodiment of the present application can calibrate the conversion relationship between the camera coordinate systems of multiple cameras.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a camera calibration method, apparatus, device, and medium. Background Art

[0002] In scenarios such as map building and positioning, in order to increase the monitoring range, multiple cameras are usually deployed. Taking a mobile robot as an example, in order to improve the field of view of the mobile robot, multiple cameras are usually deployed on the mobile robot along different directions. To facilitate unified processing of the images collected by the above multiple cameras, it is necessary to calibrate the conversion relationship between the camera coordinate systems of the above multiple cameras.

[0003] In the related art, a calibration board can be deployed in the common view area of multiple cameras. Multiple cameras perform image acquisition on the same calibration board. For each camera, based on the image of the calibration board collected, the conversion relationship between the camera coordinate system of the camera and the calibration board coordinate system of the calibration board can be calibrated, and then based on the above conversion relationship, the conversion relationship between the camera coordinate systems of each camera can be calculated.

[0004] Although the above solution can calibrate the conversion relationship between the camera coordinate systems of different cameras, when the common view area of multiple cameras is small or there is no common view area, for example, when the cameras deployed on the mobile robot face forward, side, and rear respectively, since it is difficult for the above multiple cameras to collect images of the same calibration board, the conversion relationship between the camera coordinate systems of the multiple cameras relative to the calibration board coordinate system of the same calibration board cannot be obtained, and thus it is difficult to calculate the conversion relationship between the camera coordinate systems of each camera. Summary of the Invention

[0005] The purpose of the embodiments of this application is to provide a camera calibration method, apparatus, device, and medium to calibrate the conversion relationship between the camera coordinate systems of multiple cameras. The specific technical solutions are as follows:

[0006] In a first aspect, the embodiments of this application provide a camera calibration method, and the method includes:

[0007] For at least two calibration boards among multiple calibration boards, obtain a reference image collected by a reference camera that includes the at least two calibration boards. Use the reference image to obtain the conversion relationship between the reference camera coordinate system of the reference camera and the calibration board coordinate system of each calibration board. Use the obtained conversion relationship to calculate the first conversion relationship between the calibration board coordinate systems of the at least two calibration boards, where the multiple calibration boards are respectively located in the image acquisition areas of multiple cameras to be calibrated;

[0008] According to the calculated first conversion relationship, determine the reference conversion relationship of the calibration plate coordinate system of each calibration plate relative to the reference calibration plate coordinate system of the reference calibration plate. Using the reference conversion relationship, calculate the reference coordinates of each calibration point in each calibration plate in the reference calibration plate coordinate system, where the reference calibration plate is any one of the multiple calibration plates;

[0009] For each camera to be calibrated, obtain the target image of the target calibration plate collected by the camera to be calibrated. Using the target image and the reference coordinates of each calibration point in the target calibration plate, obtain the second conversion relationship of the camera coordinate system of the camera to be calibrated relative to the reference calibration plate coordinate system, where the target calibration plate is: the calibration plate corresponding to the camera to be calibrated, and the calibration plate corresponding to each camera to be calibrated is: the calibration plate located within the image acquisition area of the camera to be calibrated;

[0010] For any two cameras to be calibrated, use the obtained second conversion relationships of the two cameras to be calibrated to calculate the target conversion relationship between the camera coordinate systems of the two cameras to be calibrated.

[0011] In an embodiment of the present application, for at least two calibration plates among the multiple calibration plates, obtain the reference image collected by the reference camera and including the at least two calibration plates. Using the reference image, obtain the conversion relationship of the reference camera coordinate system of the reference camera relative to the calibration plate coordinate system of each calibration plate. Using the obtained conversion relationship to calculate the first conversion relationship between the calibration plate coordinate systems of the at least two calibration plates, including:

[0012] For the first calibration plate and the second calibration plate among the multiple calibration plates, obtain the reference image collected by the reference camera and including the first calibration plate and the second calibration plate. Using the reference image, obtain the conversion relationship of the reference camera coordinate system of the reference camera relative to the first calibration plate coordinate system of the first calibration plate and the conversion relationship of the reference camera coordinate system relative to the second calibration plate coordinate system of the second calibration plate. Using the obtained conversion relationship to calculate the first conversion relationship between the second calibration plate coordinate system and the first calibration plate coordinate system;

[0013] Take the second calibration plate as the first calibration plate and any calibration plate that has not obtained the first conversion relationship as the second calibration plate, and return to the step of obtaining the reference image collected by the reference camera and including the first calibration plate and the second calibration plate for the first calibration plate and the second calibration plate among the multiple calibration plates until all calibration plates are traversed.

[0014] In an embodiment of the present application, the determining the reference conversion relationship of the calibration plate coordinate system of each calibration plate relative to the reference calibration plate coordinate system of the reference calibration plate according to the calculated first conversion relationship includes:

[0015] Obtain the reference images collected by the reference camera, which include the reference calibration board and the third calibration board. Using the reference images, obtain the conversion relationship between the reference camera coordinate system and the reference calibration board coordinate system of the reference calibration board, and the conversion relationship between the reference camera coordinate system and the third calibration board coordinate system of the third calibration board. Calculate the auxiliary conversion relationship between the third calibration board coordinate system and the reference calibration board coordinate system using the obtained conversion relationships, where the third calibration board is the finally determined second calibration board, and the reference calibration board is the initial first calibration board;

[0016] Correct each first conversion relationship using the auxiliary conversion relationship;

[0017] Using the corrected first conversion relationships, determine the reference conversion relationships between the calibration board coordinate systems of each calibration board and the reference calibration board coordinate system.

[0018] In one embodiment of the present application, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards;

[0019] The step of correcting each first conversion relationship using the auxiliary conversion relationship includes:

[0020] Taking reducing the loss as the principle, correct each first conversion relationship using the auxiliary conversion relationship, where the calculation method of the loss is:

[0021]

[0022] The objectionF represents the loss, and the represents the auxiliary conversion relationship, and the represents the first conversion relationship between the calibration board coordinate system of the calibration board corresponding to the i-th camera to be calibrated and the calibration board coordinate system of the calibration board corresponding to the (i - 1)-th camera to be calibrated. The N represents the number of all cameras to be calibrated, the i represents the calibration board corresponding to the i-th camera to be calibrated, and the represents the mapping transformed to the corresponding Lie algebra, and the represents the mapping transformed to the corresponding Lie algebra.

[0023] In one embodiment of the present application, for any two cameras to be calibrated, calculating the target conversion relationship between the camera coordinate systems of the two cameras to be calibrated using the obtained second conversion relationship of the two cameras to be calibrated includes:

[0024] Optimize the second conversion relationships of each camera to be calibrated;

[0025] For any two cameras to be calibrated, the target transformation relationship between the camera coordinate systems of the two cameras to be calibrated is calculated by using the optimized second transformation relationship of the two cameras to be calibrated.

[0026] In one embodiment of the present application, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards;

[0027] The optimization of the second transformation relationship of each camera to be calibrated includes:

[0028] The second transformation relationship of each camera to be calibrated is optimized according to the following formula:

[0029]

[0030]

[0031] Among them, the represents the second transformation relationship of the jth camera to be calibrated, the S represents the optimized second transformation relationship of the jth camera to be calibrated, the N represents the number of cameras to be calibrated, the j represents the jth camera to be calibrated, the M represents the number of calibration points in each calibration board, the k represents the kth calibration point, the S represents the number of target images collected by each camera to be calibrated, the s represents the s-th target image collected, the x iks represents the image coordinates of the kth calibration point in the calibration board corresponding to the ith camera to be calibrated in the s-th target image, and the X jk represents the reference coordinates of the kth calibration point in the calibration board corresponding to the jth camera to be calibrated.

[0032] In one embodiment of the present application, for any two cameras to be calibrated, the target transformation relationship between the camera coordinate systems of the two cameras to be calibrated is calculated by using the obtained second transformation relationship of the two cameras to be calibrated, including:

[0033] For the ith camera to be calibrated and the jth camera to be calibrated, the target transformation relationship between the camera coordinate systems of the ith camera to be calibrated relative to the jth camera to be calibrated is calculated according to the following formula

[0034]

[0035] Among them, the represents the second transformation relationship of the jth camera to be calibrated, and the represents the second transformation relationship of the ith camera to be calibrated.

[0036] Second aspect, an embodiment of the present application provides a camera calibration device, the device comprising:

[0037] A first relationship obtaining module, configured to, for at least two calibration plates among a plurality of calibration plates, obtain a reference image collected by a reference camera and including the at least two calibration plates, and use the reference image to obtain a conversion relationship between the reference camera coordinate system of the reference camera and the calibration plate coordinate system of each calibration plate, and calculate a first conversion relationship between the calibration plate coordinate systems of the at least two calibration plates by using the obtained conversion relationship, wherein the plurality of calibration plates are respectively located in the image acquisition regions of a plurality of cameras to be calibrated;

[0038] A reference coordinate obtaining module, configured to determine a reference conversion relationship between the calibration plate coordinate systems of each calibration plate and the reference calibration plate coordinate system of a reference calibration plate according to the calculated first conversion relationship, and calculate the reference coordinates of each calibration point in each calibration plate in the reference calibration plate coordinate system by using the reference conversion relationship, wherein the reference calibration plate is any one of the plurality of calibration plates;

[0039] A second relationship obtaining module, configured to, for each camera to be calibrated, obtain a target image of a target calibration plate collected by the camera to be calibrated, and obtain a second conversion relationship between the camera coordinate system of the camera to be calibrated and the reference calibration plate coordinate system by using the target image and the reference coordinates of each calibration point in the target calibration plate, wherein the target calibration plate is: the calibration plate corresponding to the camera to be calibrated, and the calibration plate corresponding to each camera to be calibrated is: the calibration plate located in the image acquisition region of the camera to be calibrated;

[0040] A target relationship obtaining module, configured to, for any two cameras to be calibrated, calculate a target conversion relationship between the camera coordinate systems of the two cameras to be calibrated by using the obtained second conversion relationships of the two cameras to be calibrated.

[0041] In an embodiment of the present application, the first relationship obtaining module includes:

[0042] A first relationship obtaining unit, configured to, for a first calibration plate and a second calibration plate among a plurality of calibration plates, obtain a reference image collected by the reference camera and including the first calibration plate and the second calibration plate, and use the reference image to obtain a conversion relationship between the reference camera coordinate system of the reference camera and the first calibration plate coordinate system of the first calibration plate, and a conversion relationship between the reference camera coordinate system of the reference camera and the second calibration plate coordinate system of the second calibration plate, and calculate a first conversion relationship between the second calibration plate coordinate system and the first calibration plate coordinate system by using the obtained conversion relationships;

[0043] The calibration board traversal unit is used to take the second calibration board as the first calibration board and any calibration board that has not obtained the first conversion relationship as the second calibration board, and return the steps of obtaining the reference images collected by the reference camera including the first calibration board and the second calibration board for the first calibration board and the second calibration board among the multiple calibration boards until all calibration boards are traversed.

[0044] In an embodiment of the present application, the reference coordinate obtaining module includes:

[0045] The auxiliary relationship obtaining unit is used to obtain the reference images collected by the reference camera including the reference calibration board and the third calibration board, and use the reference images to obtain the conversion relationship between the reference camera coordinate system and the reference calibration board coordinate system of the reference calibration board, and the conversion relationship between the reference camera coordinate system and the third calibration board coordinate system of the third calibration board, and calculate the auxiliary conversion relationship between the third calibration board coordinate system and the reference calibration board coordinate system by using the obtained conversion relationships, where the third calibration board is the last determined second calibration board, and the reference calibration board is the initial first calibration board;

[0046] The relationship correction unit is used to correct each first conversion relationship by using the auxiliary conversion relationship;

[0047] The reference relationship obtaining unit is used to determine the reference conversion relationship between the calibration board coordinate systems of each calibration board and the reference calibration board coordinate system by using each corrected first conversion relationship;

[0048] The reference coordinate obtaining unit is used to calculate the reference coordinates of each calibration point in each calibration board in the reference calibration board coordinate system by using the reference conversion relationship.

[0049] In an embodiment of the present application, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards;

[0050] The relationship correction unit is specifically used for:

[0051] Taking reducing the loss as the principle, correct each first conversion relationship by using the auxiliary conversion relationship, where the calculation method of the loss is:

[0052]

[0053] The objectionF represents the loss, and the represents the auxiliary conversion relationship, and the Represents: the first transformation relationship between the calibration board coordinate systems of the calibration boards corresponding to the i-th camera to be calibrated relative to the calibration board coordinate systems of the calibration boards corresponding to the (i - 1)-th camera to be calibrated, where N represents the number of all cameras to be calibrated, i represents the calibration board corresponding to the i-th camera to be calibrated, and the represents the mapping that transforms to the corresponding Lie algebra, and the represents the mapping that transforms to the corresponding Lie algebra.

[0054] In an embodiment of the present application, the target relationship obtaining module includes:

[0055] A relationship optimization unit for optimizing the second transformation relationships of each camera to be calibrated;

[0056] A target relationship obtaining unit for calculating the target transformation relationship between the camera coordinate systems of any two cameras to be calibrated by using the optimized second transformation relationships of the two cameras to be calibrated.

[0057] In an embodiment of the present application, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards;

[0058] The relationship optimization unit is specifically used for:

[0059] Optimizing the second transformation relationships of each camera to be calibrated according to the following formula:

[0060]

[0061]

[0062] where the represents the second transformation relationship of the j-th camera to be calibrated, S represents the optimized second transformation relationship of the j-th camera to be calibrated, N represents the number of cameras to be calibrated, j represents the j-th camera to be calibrated, M represents the number of calibration points in each calibration board, k represents the k-th calibration point, S represents the number of target images collected by each camera to be calibrated, s represents the s-th target image collected, and x iks represents the image coordinate of the k-th calibration point in the calibration board corresponding to the i-th camera to be calibrated in the s-th target image, and X jk represents the reference coordinate of the k-th calibration point in the calibration board corresponding to the j-th camera to be calibrated.

[0063] In an embodiment of the present application, the target relationship obtaining module is specifically used for:

[0064] For the i-th camera to be calibrated and the j-th camera to be calibrated, calculate the target transformation relationship between the camera coordinate systems of the i-th camera to be calibrated and the j-th camera to be calibrated according to the following formula

[0065]

[0066] wherein, the represents the second transformation relationship of the j-th camera to be calibrated, and the represents the second transformation relationship of the i-th camera to be calibrated.

[0067] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0068] The memory is used to store a computer program;

[0069] The processor is configured to implement the method steps of any one of the first aspects when executing the program stored on the memory.

[0070] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the method steps of any one of the first aspects are implemented.

[0071] The embodiment of the present application also provides a computer program product containing instructions, which when running on a computer, causes the computer to execute the camera calibration method described above.

[0072] Advantageous effects of the embodiment of the present application:

[0073] In the camera calibration solution provided by the embodiments of the present application, first, the first conversion relationship between the calibration board coordinate systems of multiple calibration boards can be obtained by calibrating with a reference camera. Then, according to this first conversion relationship, the coordinates of the calibration points in each calibration board are uniformly converted to the same reference calibration board coordinate system, and the reference coordinates of different calibration points in different calibration boards in the same coordinate system are obtained. Then, for different cameras to be calibrated, the second conversion relationship between the camera coordinate system of the camera to be calibrated and the calibration board coordinate system is respectively obtained by using the reference coordinates of the calibration points in different calibration boards. Since the reference coordinates of the calibration points in different calibration boards are the coordinates in the same reference calibration board coordinate system, the above-mentioned second conversion relationship can be understood as: the conversion relationship between the camera coordinate systems of different cameras to be calibrated and the same reference calibration board coordinate system. In view of the fact that the second conversion relationships of different cameras to be calibrated are established based on the same reference calibration board, the target conversion relationship between the camera coordinate systems of any two cameras to be calibrated can be directly calculated according to the above-mentioned second conversion relationship. It can be seen that by applying the solution provided by the embodiments of the present application, the conversion relationship between the camera coordinate systems of multiple cameras can be calibrated. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0075] Figure 1 It is a schematic flowchart of a camera calibration method provided by an embodiment of the present application;

[0076] Figure 2 It is a schematic flowchart of another camera calibration method provided by an embodiment of the present application;

[0077] Figure 3 It is a schematic flowchart of a method for obtaining a reference conversion relationship provided by an embodiment of the present application;

[0078] Figure 4 It is a schematic flowchart of yet another camera calibration method provided by an embodiment of the present application;

[0079] Figure 5 It is a schematic structural diagram of a camera calibration device provided by an embodiment of the present application;

[0080] Figure 6 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0081] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0082] In order to calibrate the conversion relationship between the camera coordinate systems of multiple cameras, the embodiments of the present application provide a camera calibration method, device, equipment and medium, which will be introduced in detail below.

[0083] The embodiments of the present application provide a camera calibration method. This method can be applied to electronic devices such as computers, mobile phones, servers, tablet computers, NVRs (Network Video Recorder), etc. The method includes:

[0084] For at least two calibration plates among multiple calibration plates, obtain a reference image collected by a reference camera that includes at least two calibration plates. Using the reference image, obtain the conversion relationship between the reference camera coordinate system of the reference camera and the calibration plate coordinate system of each calibration plate. Calculate the first conversion relationship between the calibration plate coordinate systems of at least two calibration plates using the obtained conversion relationship. Among them, the multiple calibration plates are respectively located in the image acquisition areas of multiple cameras to be calibrated;

[0085] According to the calculated first conversion relationship, determine the reference conversion relationship between the calibration plate coordinate systems of each calibration plate and the reference calibration plate coordinate system of the reference calibration plate. Using the reference conversion relationship, calculate the reference coordinates of each calibration point in each calibration plate in the reference calibration plate coordinate system. Among them, the reference calibration plate is any one of the multiple calibration plates;

[0086] For each camera to be calibrated, obtain a target image of the target calibration plate collected by the camera to be calibrated. Using the target image and the reference coordinates of each calibration point in the target calibration plate, obtain the second conversion relationship between the camera coordinate system of the camera to be calibrated and the reference calibration plate coordinate system. Among them, the target calibration plate is: the calibration plate corresponding to the camera to be calibrated, and the calibration plate corresponding to each camera to be calibrated is: the calibration plate located in the image acquisition area of the camera to be calibrated;

[0087] For any two cameras to be calibrated, calculate the target conversion relationship between the camera coordinate systems of the two cameras to be calibrated using the obtained second conversion relationships of the two cameras to be calibrated.

[0088] In the camera calibration scheme provided in the above embodiments, first, the first conversion relationship between the calibration board coordinate systems of multiple calibration boards can be obtained by calibrating with a reference camera. Then, according to this first conversion relationship, the coordinates of the marked points in each calibration board are uniformly converted to the same reference calibration board coordinate system, and the reference coordinates of different marked points in different calibration boards in the same coordinate system are obtained. Then, for different cameras to be calibrated, the second conversion relationship between the camera coordinate system of the camera to be calibrated and the calibration board coordinate system is respectively obtained by using the reference coordinates of the marked points in different calibration boards. Since the reference coordinates of the marked points in different calibration boards are the coordinates in the same reference calibration board coordinate system, the above second conversion relationship can be understood as: the conversion relationship between the camera coordinate systems of different cameras to be calibrated and the same reference calibration board coordinate system. In view of the fact that the second conversion relationships of different cameras to be calibrated are established based on the same reference calibration board, the target conversion relationship between the camera coordinate systems of any two cameras to be calibrated can be directly calculated according to the above second conversion relationship. It can be seen that by applying the scheme provided in the above embodiments, the conversion relationship between the camera coordinate systems of multiple cameras can be calibrated.

[0089] The above camera calibration method will be introduced in detail below.

[0090] See Figure 1 , Figure 1 which is a schematic flowchart of a camera calibration method provided in an embodiment of the present application. The method includes the following steps S101 - S104:

[0091] S101. For at least two of multiple calibration boards, obtain a reference image collected by a reference camera that includes at least two calibration boards. Use the reference image to obtain the conversion relationship between the reference camera coordinate system of the reference camera and the calibration board coordinate system of each calibration board, and calculate the first conversion relationship between the calibration board coordinate systems of at least two calibration boards by using the obtained conversion relationship.

[0092] Among them, multiple calibration boards are respectively located in the image acquisition areas of multiple cameras to be calibrated. Each calibration board contains multiple marked points. The above calibration board can be a checkerboard calibration board, etc.

[0093] The reference camera is a camera whose internal parameters have been calibrated in advance. The above internal parameters refer to the conversion relationship between the three-dimensional coordinates in the camera coordinate system and the image coordinates in the image collected by the camera. The internal parameters of the reference camera can be calibrated by using the Zhang's calibration method.

[0094] The calibration board coordinate system of each calibration board refers to: a three-dimensional coordinate system established based on this calibration board. For example, it can be: a right-handed coordinate system with the long side of the calibration board as the X axis, the short side as the Y axis, and the plane where it is located as z = 0.

[0095] The coordinate system of each camera refers to: a three-dimensional coordinate system established based on the camera. For example, it can be: a left-handed coordinate system or a right-handed coordinate system established with the optical center of the camera as the coordinate origin, the direction of the long side of the acquired image as the X-axis, the direction of the short side of the acquired image as the Y-axis, and the direction perpendicular to the plane of the acquired image as the Z-axis.

[0096] Specifically, for at least two calibration plates among multiple calibration plates, the reference camera can be used to acquire images of the at least two calibration plates as reference images. Then, for any one of the calibration plates, based on the internal parameters of the reference camera obtained in advance, the coordinates of each calibration point in the calibration plate corresponding to the calibration plate coordinate system, and the image coordinates of each calibration point of the calibration plate in the image, the conversion relationship between the reference camera coordinate system of the reference camera and the calibration plate coordinate system of the calibration plate can be calculated. That is to say, for each of the above at least two calibration plates, the conversion relationship between the reference camera coordinate system of the reference camera and the calibration plate coordinate systems of each calibration plate can be obtained. Subsequently, the conversion relationship between the calibration plate coordinate systems of each calibration plate can be further calculated using the above conversion relationship as the first conversion relationship until all calibration plates are traversed.

[0097] In an embodiment of the present application, when calculating the conversion relationship between the reference camera coordinate system and the calibration plate coordinate system, based on the internal parameters of the reference camera, the coordinates of each calibration point in the calibration plate coordinate system, and the image coordinates of each calibration point of the calibration plate in the image, the Perspective-n-Point (PNP) algorithm can be used to calculate the conversion relationship between the reference camera coordinate system of the reference camera and the calibration plate coordinate system of the calibration plate. In addition, it can also be calibrated using other algorithms, and the embodiments of the present application do not limit this.

[0098] In an embodiment of the present application, the first conversion relationship between the calibration plate coordinate systems of two calibration plates can be calculated using the following formula:

[0099]

[0100] Among them, represents the first conversion relationship between the calibration plate coordinate systems of the i-th calibration plate and the (i + 1)-th calibration plate, represents the conversion relationship between the reference camera coordinate system of the reference camera and the calibration plate coordinate system of the i-th calibration plate; represents the conversion relationship between the reference camera coordinate system of the reference camera and the calibration plate coordinate system of the (i + 1)-th calibration plate.

[0101] In one embodiment of the present application, the number of calibration plates may be the same as the number of cameras to be calibrated, or on the basis of ensuring that at least one complete calibration plate is included within the image acquisition range of each camera to be calibrated, the number of calibration plates may be controlled to be less than the number of cameras to be calibrated. In this case, there is at least one calibration plate that is at least simultaneously within the image acquisition ranges of two cameras to be calibrated.

[0102] S102. According to the calculated first conversion relationship, determine the reference conversion relationship of the calibration plate coordinate system of each calibration plate relative to the reference calibration plate coordinate system of the reference calibration plate. Using the reference conversion relationship, calculate the reference coordinates of each calibration point in each calibration plate in the reference calibration plate coordinate system.

[0103] Among them, the reference calibration plate is any one of the multiple calibration plates.

[0104] Specifically, any one calibration plate can be selected from the above multiple calibration plates as the reference calibration plate. For each calibration plate other than the reference calibration plate, the conversion relationship between the calibration plate coordinate system of this calibration plate and the reference calibration plate coordinate system of the above reference calibration plate can be calculated using the first conversion relationship obtained in the above step S101 as the reference conversion relationship. Then, using this reference conversion relationship, the coordinates of each calibration point in this calibration plate in the calibration plate coordinate system are converted to obtain the coordinates of each calibration point in this calibration plate in the above reference calibration plate coordinate system as the reference coordinates.

[0105] For example, assume that there are 4 calibration plates. The first conversion relationship obtained in the above step S101 includes: the conversion relationship q1 between the calibration plate coordinate system f1 of the first calibration plate and the calibration plate coordinate system f2 of the second calibration plate, the conversion relationship q2 between the calibration plate coordinate system f2 of the second calibration plate and the calibration plate coordinate system f3 of the third calibration plate, and the conversion relationship q3 between the calibration plate coordinate system f3 of the third calibration plate and the calibration plate coordinate system f4 of the fourth calibration plate. Select the first calibration plate as the reference calibration plate, and the calibration plate coordinate system f1 as the reference calibration plate coordinate system;

[0106] For the second calibration plate, the coordinates of the calibration points in the calibration plate coordinate system f2 of this second calibration plate can be multiplied by the above conversion relationship q1 to obtain the coordinates of the calibration points in the second calibration plate in the reference calibration plate coordinate system f1;

[0107] For the third calibration plate, q1*q2 can be calculated as the conversion relationship between the calibration plate coordinate system f1 and the calibration plate coordinate system f3. The coordinates of the calibration points in the calibration plate coordinate system f3 of this third calibration plate can be multiplied by the above conversion relationship q1*q2 to obtain the coordinates of the calibration points in the third calibration plate in the reference calibration plate coordinate system f1;

[0108] For the fourth calibration board, q1*q2*q3 can be calculated as the conversion relationship between the calibration board coordinate system f1 and the calibration board coordinate system f4. Multiply the coordinates of the marked points in the calibration board coordinate system f4 of the fourth calibration board by the above conversion relationship q1*q2*q3 to obtain the coordinates of the marked points in the fourth calibration board in the reference calibration board coordinate system f1.

[0109] In an embodiment of the present application, the positions of each calibration board can be kept fixed to ensure that the relative position relationship of each calibration board remains unchanged. In this case, after obtaining the reference coordinates of each marked point in each calibration board in the reference calibration board coordinate system, the above reference coordinates can be stored for subsequent reuse, so that when performing camera calibration subsequently, there is no need to repeatedly calculate the reference coordinates of each marked point in each calibration board in the reference calibration board coordinate system, saving computing resources and improving the camera calibration efficiency.

[0110] S103. For each camera to be calibrated, obtain the target image of the target calibration board collected by the camera to be calibrated, and use the target image and the reference coordinates of each marked point in the target calibration board to obtain the second conversion relationship between the camera coordinate system of the camera to be calibrated and the reference calibration board coordinate system.

[0111] Wherein, the target calibration board is the calibration board corresponding to the camera to be calibrated, and the calibration board corresponding to each camera to be calibrated is the calibration board located within the image acquisition area of the camera to be calibrated.

[0112] The internal parameters of each camera to be calibrated have been pre-calibrated. The above internal parameters refer to the conversion relationship between the three-dimensional coordinates in the camera coordinate system of the camera to be calibrated and the image coordinates in the image collected by the camera to be calibrated. The internal parameters of the camera to be calibrated can be calibrated using the Zhang's calibration method.

[0113] Specifically, for each camera to be calibrated, a calibration board located within the image acquisition area of the camera to be calibrated can be used as the target calibration board, and the image of the above target calibration board is collected by the camera to be calibrated as the target image. Then, the image coordinates of each marked point on the target calibration board in the image are detected. Using the above image coordinates, the reference coordinates of each marked point in the target calibration board, and the internal parameters of the camera to be calibrated, the conversion relationship between the camera coordinate system of the camera to be calibrated and the calibration board coordinate system of the target calibration board can be calculated. Since the above reference coordinates are the coordinates of each marked point in the target calibration board in the reference calibration board coordinate system, the above conversion relationship is the target conversion relationship between the camera coordinate system of the camera to be calibrated and the reference calibration board coordinate system.

[0114] In one embodiment of the present application, when calculating the target transformation relationship between the camera coordinate system of the camera to be calibrated and the reference calibration board coordinate system, based on the internal parameters of the camera to be calibrated, the reference coordinates of each calibration point in the target calibration board in the reference calibration board coordinate system, and the image coordinates of each calibration point of the target calibration board in the image, the transformation relationship between the camera coordinate system of the camera to be calibrated and the reference calibration board coordinate system can be calculated using the PNP algorithm. In addition, it can also be calibrated using other algorithms, and the embodiments of the present application do not limit this.

[0115] S104. For any two cameras to be calibrated, use the obtained second transformation relationship of these two cameras to be calibrated to calculate the target transformation relationship between the camera coordinate systems of these two cameras to be calibrated.

[0116] Specifically, between any two cameras to be calibrated, the obtained second transformation relationship of these two cameras to be calibrated can be used to calculate the target transformation relationship between the camera coordinate systems of these two cameras to be calibrated.

[0117] In one embodiment of the present application, for the i-th camera to be calibrated and the j-th camera to be calibrated, the target transformation relationship between the camera coordinate systems of the i-th camera to be calibrated relative to the j-th camera to be calibrated can be calculated according to the following formula

[0118]

[0119] where represents the second transformation relationship of the j-th camera to be calibrated, represents the second transformation relationship of the i-th camera to be calibrated.

[0120] In one embodiment of the present application, the second transformation relationship of each camera to be calibrated can be optimized; for any two cameras to be calibrated, use the optimized second transformation relationship of these two cameras to be calibrated to calculate the target transformation relationship between the camera coordinate systems of these two cameras to be calibrated.

[0121] Specifically, there may be errors in the obtained second transformation relationships of each camera to be calibrated in the above step S103. In order to improve the accuracy of camera calibration, the second transformation relationships of each camera to be calibrated can be first optimized using a preset optimization algorithm, and then based on the optimized second transformation relationships, the target transformation relationships between the camera coordinate systems of any two cameras to be calibrated are calculated.

[0122] In one embodiment of the present application, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards; on this basis, the second transformation relationships of each camera to be calibrated can be optimized according to the following formula:

[0123]

[0124]

[0125] Among them, the represents the second conversion relationship of the j-th camera to be calibrated, the S represents the optimized second conversion relationship of the j-th camera to be calibrated, the N represents the number of cameras to be calibrated, the j represents the j-th camera to be calibrated, the M represents the number of calibration points in each calibration board, the k represents the k-th calibration point, the S represents the number of images of the target images collected by each camera to be calibrated, the s represents the s-th target image collected, and the x iks represents the image coordinates of the k-th calibration point in the calibration board corresponding to the i-th camera to be calibrated in the s-th target image, and the X jk represents the reference coordinates of the k-th calibration point in the calibration board corresponding to the j-th camera to be calibrated.

[0126] In the camera calibration scheme provided in the above embodiment, first, the first conversion relationship between the calibration board coordinate systems of multiple calibration boards can be obtained by using a reference camera. Then, according to this first conversion relationship, the coordinates of the calibration points in each calibration board are uniformly converted to the same reference calibration board coordinate system, and the reference coordinates of different calibration points in different calibration boards in the same coordinate system are obtained. Then, for different cameras to be calibrated, the second conversion relationship between the camera coordinate system of the camera to be calibrated and the calibration board coordinate system is obtained by using the reference coordinates of the calibration points in different calibration boards respectively. Since the reference coordinates of the calibration points in different calibration boards are the coordinates in the same reference calibration board coordinate system, the above second conversion relationship can be understood as: the conversion relationship between the camera coordinate systems of different cameras to be calibrated relative to the same reference calibration board coordinate system. In view of the fact that the second conversion relationships of different cameras to be calibrated are established based on the same reference calibration board, the target conversion relationship between the camera coordinate systems of any two cameras to be calibrated can be directly calculated according to the above second conversion relationship. It can be seen that by applying the scheme provided in the above embodiment, the conversion relationship between the camera coordinate systems of multiple cameras can be calibrated.

[0127] See Figure 2 , Figure 2 which is a schematic flowchart of another camera calibration method provided by the embodiment of the present application. For the above step S101 to obtain the first conversion relationship, the following steps S1011-S1012 may be included:

[0128] S1011. For the first calibration board and the second calibration board among multiple calibration boards, obtain a reference image collected by a reference camera and containing the first calibration board and the second calibration board. Using the reference image, obtain the conversion relationship between the reference camera coordinate system of the reference camera and the first calibration board coordinate system of the first calibration board, and the conversion relationship between the reference camera coordinate system of the reference camera and the second calibration board coordinate system of the second calibration board. Calculate the first conversion relationship between the second calibration board coordinate system and the first calibration board coordinate system using the obtained conversion relationships.

[0129] Specifically, two calibration boards can be selected from multiple calibration boards and used as the first calibration board and the second calibration board respectively. The first calibration board and the second calibration board can be calibration boards with close distances or the same orientation, etc., so that both the first calibration board and the second calibration board are within the image acquisition range of the reference camera.

[0130] The reference camera can be used to collect an image containing both the first calibration board and the second calibration board as the reference image. Then, for the first calibration board, based on the internal parameters of the reference camera, the coordinates of each calibration point in the first calibration board coordinate system corresponding to the first calibration board, and the image coordinates of each calibration point of the first calibration board in the image, the conversion relationship g1 between the reference camera coordinate system of the reference camera and the first calibration board coordinate system can be calculated. And for the second calibration board, based on the internal parameters of the reference camera, the coordinates of each calibration point in the second calibration board coordinate system corresponding to the second calibration board, and the image coordinates of each calibration point of the second calibration board in the image, the conversion relationship g2 between the reference camera coordinate system of the reference camera and the second calibration board coordinate system can be calculated. Subsequently, the first conversion relationship between the first calibration board coordinate system and the second calibration board coordinate system can be further calculated using the above conversion relationships g1 and g2.

[0131] S1012. Use the second calibration board as the first calibration board and any calibration board for which the first conversion relationship has not been obtained as the second calibration board, and return to step S1011 until all calibration boards are traversed.

[0132] Specifically, after obtaining the first conversion relationship between the first calibration board coordinate system and the second calibration board coordinate system, the second calibration board can be used as the new first calibration board. Then, select a calibration board from the calibration boards for which the first conversion relationship has not been obtained as the new second calibration board. The selected calibration board can be adjacent to or have the same orientation as the new first calibration board. Then return to the above step S1011 to obtain the first conversion relationship between the first calibration board coordinate system corresponding to the new first calibration board and the second calibration board coordinate system corresponding to the new second calibration board until all calibration boards are traversed, and finally obtain the first conversion relationship between the calibration board coordinate systems of every two calibration boards.

[0133] Based on the above solution, in the embodiment of the present application, when obtaining the reference conversion relationship in the above step S102, a method for obtaining the reference conversion relationship is provided, which will be introduced in detail below.

[0134] See Figure 3 , Figure 3 which is a schematic flowchart of a method for obtaining a reference conversion relationship provided by an embodiment of the present application, including the following steps S301 - S303:

[0135] S301, obtain a reference image collected by a reference camera, which includes a reference calibration board and a third calibration board. Using the reference image, obtain the conversion relationship between the reference camera coordinate system and the reference calibration board coordinate system of the reference calibration board, and the conversion relationship between the reference camera coordinate system and the third calibration board coordinate system of the third calibration board. Calculate the auxiliary conversion relationship between the third calibration board coordinate system and the reference calibration board coordinate system using the obtained conversion relationships.

[0136] Among them, the third calibration board is the finally determined second calibration board, and the reference calibration board is the initial first calibration board.

[0137] Specifically, the last second calibration board can be used as the third calibration board, and the initial first calibration board can be used as the reference calibration board. After traversing each calibration board according to the above steps S1011 - S1012, use the reference camera to collect the images of the above reference calibration board and the third calibration board again as the reference image, and then use the reference image to obtain the conversion relationship between the third calibration board coordinate system and the reference calibration board coordinate system as the auxiliary conversion relationship.

[0138] S302, correct each first conversion relationship using the auxiliary conversion relationship.

[0139] Specifically, in theory, using the first conversion relationship between the calibration board coordinate systems of two calibration boards, the conversion relationship between the reference calibration board coordinate system and the third calibration board coordinate system can be calculated, and this conversion relationship is consistent with the above auxiliary conversion relationship. However, due to calculation errors and other reasons, there may be a difference between the calculated conversion relationship and the above auxiliary conversion relationship. Based on this difference, each first conversion relationship can be corrected to improve the accuracy of the first conversion relationship.

[0140] In an embodiment of the present application, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards. On this basis, taking reducing loss as the principle, each first conversion relationship can be corrected using the auxiliary conversion relationship, where the calculation method of the loss is:

[0141]

[0142] Among them, the above-mentioned objectionF represents loss. represents the auxiliary conversion relationship. represents: the first conversion relationship between the calibration board coordinate system of the calibration board corresponding to the i-th camera to be calibrated and the calibration board coordinate system of the calibration board corresponding to the (i - 1)-th camera to be calibrated. N represents the number of all cameras to be calibrated, and i represents the calibration board corresponding to the i-th camera to be calibrated.

[0143] S303. Using each of the corrected first conversion relationships, determine the reference conversion relationship between the calibration board coordinate system of each calibration board and the reference calibration board coordinate system.

[0144] Specifically, after obtaining each of the corrected first conversion relationships, based on the calibration board coordinate system of the above-mentioned corrected calibration board, the reference conversion relationship corresponding to each calibration board can be calculated.

[0145] See Figure 4 , Figure 4 which is a schematic flowchart of another camera calibration method provided by an embodiment of the present application. The method includes the following steps S401 - S409:

[0146] S401. For the first calibration board and the second calibration board among multiple calibration boards, obtain the reference image collected by the reference camera and including the first calibration board and the second calibration board. Using the reference image, obtain the conversion relationship between the reference camera coordinate system of the reference camera and the first calibration board coordinate system of the first calibration board, and the conversion relationship between the reference camera coordinate system of the reference camera and the second calibration board coordinate system of the second calibration board. Calculate the first conversion relationship between the second calibration board coordinate system and the first calibration board coordinate system using the obtained conversion relationships.

[0147] Among them, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards.

[0148] S402. Take the second calibration board as the first calibration board, and take any calibration board that has not obtained the first conversion relationship as the second calibration board, and return to step S401 until all calibration boards are traversed.

[0149] S403. Obtain the reference image collected by the reference camera and including the reference calibration board and the third calibration board. Using the reference image, obtain the conversion relationship between the reference camera coordinate system of the reference camera and the reference calibration board coordinate system of the reference calibration board, and the conversion relationship between the reference camera coordinate system of the reference camera and the third calibration board coordinate system of the third calibration board. Calculate the auxiliary conversion relationship between the third calibration board coordinate system and the reference calibration board coordinate system using the obtained conversion relationships.

[0150] Among them, the third calibration board is: the finally determined second calibration board, and the reference calibration board is the initial first calibration board.

[0151] S404. Calibrate each first conversion relationship using the auxiliary conversion relationship.

[0152] S405. Using the calibrated first conversion relationships, determine the reference conversion relationships of the calibration plate coordinate systems of each calibration plate relative to the reference calibration plate coordinate system.

[0153] S406. Using the reference conversion relationships, calculate the reference coordinates of each calibration point in each calibration plate in the reference calibration plate coordinate system.

[0154] S407. For each camera to be calibrated, obtain the target image of the target calibration plate collected by the camera to be calibrated. Using the target image and the reference coordinates of each calibration point in the target calibration plate, obtain the second conversion relationship of the camera coordinate system of the camera to be calibrated relative to the reference calibration plate coordinate system.

[0155] Wherein, the target calibration plate is: the calibration plate corresponding to the camera to be calibrated, and the calibration plate corresponding to each camera to be calibrated is: the calibration plate located within the image acquisition area of the camera to be calibrated.

[0156] S408. Optimize the second conversion relationships of each camera to be calibrated.

[0157] S409. For any two cameras to be calibrated, using the optimized second conversion relationships of the two cameras to be calibrated, calculate the target conversion relationship between the camera coordinate systems of the two cameras to be calibrated.

[0158] In the camera calibration scheme provided in the above embodiments, the first conversion relationships between the calibration plate coordinate systems of multiple calibration plates can be obtained by first calibrating using a reference camera. Then, according to the first conversion relationships, the coordinates of the calibration points in each calibration plate are uniformly converted to the same reference calibration plate coordinate system, and the reference coordinates of different calibration points in different calibration plates in the same coordinate system are obtained. Then, for different cameras to be calibrated, the second conversion relationships of the camera coordinate systems of the cameras to be calibrated relative to the calibration plate coordinate systems are respectively calibrated using the reference coordinates of the calibration points in different calibration plates. Since the reference coordinates of the calibration points in different calibration plates are coordinates in the same reference calibration plate coordinate system, the above second conversion relationships can be understood as: the conversion relationships of the camera coordinate systems of different cameras to be calibrated relative to the same reference calibration plate coordinate system. In view of the fact that the second conversion relationships of different cameras to be calibrated are established based on the same reference calibration plate, the target conversion relationship between the camera coordinate systems of any two cameras to be calibrated can be directly calculated according to the above second conversion relationships. Thus, it can be seen that by applying the scheme provided in the above embodiments, the conversion relationships between the camera coordinate systems of multiple cameras can be calibrated.

[0159] Corresponding to the above camera calibration method, an embodiment of the present application also provides a camera calibration device, which will be introduced in detail below.

[0160] See Figure 5 , Figure 5 which is a schematic structural diagram of a camera calibration device provided by an embodiment of the present application. The device includes:

[0161] A first relationship obtaining module 501, configured to, for at least two calibration plates among a plurality of calibration plates, obtain a reference image collected by a reference camera and including the at least two calibration plates, use the reference image to obtain a conversion relationship between the reference camera coordinate system of the reference camera and the calibration plate coordinate system of each calibration plate, and calculate a first conversion relationship between the calibration plate coordinate systems of the at least two calibration plates by using the obtained conversion relationship, wherein the plurality of calibration plates are respectively located in the image acquisition areas of a plurality of cameras to be calibrated;

[0162] A reference coordinate obtaining module 502, configured to determine a reference conversion relationship between the calibration plate coordinate system of each calibration plate and the reference calibration plate coordinate system of a reference calibration plate according to the calculated first conversion relationship, and calculate the reference coordinates of each calibration point in each calibration plate in the reference calibration plate coordinate system by using the reference conversion relationship, wherein the reference calibration plate is any one of the plurality of calibration plates;

[0163] A second relationship obtaining module 503, configured to, for each camera to be calibrated, obtain a target image of a target calibration plate collected by the camera to be calibrated, and obtain a second conversion relationship between the camera coordinate system of the camera to be calibrated and the reference calibration plate coordinate system by using the target image and the reference coordinates of each calibration point in the target calibration plate, wherein the target calibration plate is: the calibration plate corresponding to the camera to be calibrated, and the calibration plate corresponding to each camera to be calibrated is: the calibration plate located in the image acquisition area of the camera to be calibrated;

[0164] A target relationship obtaining module 504, configured to, for any two cameras to be calibrated, calculate a target conversion relationship between the camera coordinate systems of the two cameras to be calibrated by using the obtained second conversion relationships of the two cameras to be calibrated.

[0165] In an embodiment of the present application, the first relationship obtaining module 501 includes:

[0166] A first relationship obtaining unit is configured to obtain, for a first calibration board and a second calibration board among a plurality of calibration boards, a reference image collected by the reference camera and including the first calibration board and the second calibration board, and use the reference image to obtain a conversion relationship between the reference camera coordinate system of the reference camera and the first calibration board coordinate system of the first calibration board, and a conversion relationship between the reference camera coordinate system of the reference camera and the second calibration board coordinate system of the second calibration board, and calculate a first conversion relationship between the second calibration board coordinate system and the first calibration board coordinate system by using the obtained conversion relationships;

[0167] A calibration board traversing unit is configured to use the second calibration board as the first calibration board and any calibration board for which the first conversion relationship has not been obtained as the second calibration board, and return to the step of obtaining, for the first calibration board and the second calibration board among the plurality of calibration boards, the reference image collected by the reference camera and including the first calibration board and the second calibration board, until all calibration boards are traversed.

[0168] In an embodiment of the present application, the reference coordinate obtaining module 502 includes:

[0169] An auxiliary relationship obtaining unit is configured to obtain a reference image collected by the reference camera and including a reference calibration board and a third calibration board, and use the reference image to obtain a conversion relationship between the reference camera coordinate system of the reference camera and the reference calibration board coordinate system of the reference calibration board, and a conversion relationship between the reference camera coordinate system of the reference camera and the third calibration board coordinate system of the third calibration board, and calculate an auxiliary conversion relationship between the third calibration board coordinate system and the reference calibration board coordinate system by using the obtained conversion relationships, where the third calibration board is the last determined second calibration board, and the reference calibration board is the initial first calibration board;

[0170] A relationship correction unit is configured to correct each first conversion relationship by using the auxiliary conversion relationship;

[0171] A reference relationship obtaining unit is configured to determine a reference conversion relationship between the calibration board coordinate systems of each calibration board and the reference calibration board coordinate system by using each corrected first conversion relationship;

[0172] A reference coordinate obtaining unit is configured to calculate the reference coordinates of each calibration point in each calibration board in the reference calibration board coordinate system by using the reference conversion relationship.

[0173] In an embodiment of the present application, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards;

[0174] The relationship correction unit is specifically configured to:

[0175] Taking the principle of reducing losses, the auxiliary conversion relationship is used to correct each first conversion relationship, where the calculation method of the loss is as follows:

[0176]

[0177] The objectionF represents the loss, and the represents the auxiliary conversion relationship, and the represents: the first conversion relationship between the calibration board coordinate systems of the calibration boards corresponding to the i-th camera to be calibrated and the calibration boards corresponding to the (i - 1)-th camera to be calibrated. The N represents the number of all cameras to be calibrated, the i represents the calibration board corresponding to the i-th camera to be calibrated, and the represents the mapping to the corresponding Lie algebra, and the represents the mapping to the corresponding Lie algebra.

[0178] In an embodiment of the present application, the target relationship obtaining module 504 includes:

[0179] A relationship optimization unit for optimizing the second conversion relationship of each camera to be calibrated;

[0180] A target relationship obtaining unit for calculating the target conversion relationship between the camera coordinate systems of any two cameras to be calibrated by using the optimized second conversion relationships of the two cameras to be calibrated.

[0181] In an embodiment of the present application, each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards;

[0182] The relationship optimization unit is specifically used for:

[0183] Optimizing the second conversion relationship of each camera to be calibrated according to the following formula:

[0184]

[0185]

[0186] Wherein, the represents the second conversion relationship of the j-th camera to be calibrated, the S represents the optimized second conversion relationship of the j-th camera to be calibrated, the N represents the number of cameras to be calibrated, the j represents the j-th camera to be calibrated, the M represents the number of calibration points in each calibration board, the k represents the k-th calibration point, the S represents the number of target images collected by each camera to be calibrated, the s represents the s-th target image collected, and the x iksdenotes the image coordinates of the k-th calibration point in the calibration board corresponding to the i-th camera to be calibrated in the s-th target image, where X jk denotes the reference coordinates of the k-th calibration point in the calibration board corresponding to the j-th camera to be calibrated.

[0187] In one embodiment of the present application, the target relationship obtaining module 504 is specifically configured to:

[0188] For the i-th camera to be calibrated and the j-th camera to be calibrated, calculate the target conversion relationship between the camera coordinate systems of the i-th camera to be calibrated relative to the j-th camera to be calibrated according to the following formula

[0189]

[0190] wherein, the denotes the second conversion relationship of the j-th camera to be calibrated, and the denotes the second conversion relationship of the i-th camera to be calibrated.

[0191] In the camera calibration scheme provided in the above embodiment, the first conversion relationship between the calibration board coordinate systems of multiple calibration boards can be obtained by using the reference camera for calibration first. Then, according to this first conversion relationship, the coordinates of the calibration points in each calibration board are uniformly converted to the same reference calibration board coordinate system, and the reference coordinates of different calibration points in different calibration boards in the same coordinate system are obtained. Then, for different cameras to be calibrated, the second conversion relationship between the camera coordinate system of the camera to be calibrated and the calibration board coordinate system is obtained by using the reference coordinates of the calibration points in different calibration boards respectively. Since the reference coordinates of the calibration points in different calibration boards are coordinates in the same reference calibration board coordinate system, the above second conversion relationship can be understood as: the conversion relationship between the camera coordinate systems of different cameras to be calibrated relative to the same reference calibration board coordinate system. Given that the second conversion relationships of different cameras to be calibrated are established based on the same reference calibration board, the target conversion relationship between the camera coordinate systems of any two cameras to be calibrated can be directly calculated according to the above second conversion relationship. Thus, it can be seen that by applying the scheme provided in the above embodiment, the conversion relationship between the camera coordinate systems of multiple cameras can be calibrated.

[0192] The embodiment of the present application further provides an electronic device, as Figure 6 shown, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604. Among them, the processor 601, the communication interface 602, and the memory 603 complete communication with each other through the communication bus 604.

[0193] The memory 603 is used to store a computer program;

[0194] The processor 601 is configured to implement the steps of the camera calibration method when executing the program stored in the memory 603.

[0195] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0196] The communication interface is used for communication between the above electronic device and other devices.

[0197] The memory may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0198] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0199] In another embodiment provided by the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above camera calibration methods are implemented.

[0200] In another embodiment provided by the present application, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the camera calibration methods in the above embodiments.

[0201] In the camera calibration scheme provided by the above embodiments, first, the first conversion relationship between the calibration board coordinate systems of multiple calibration boards can be obtained by using a reference camera for calibration. Then, according to this first conversion relationship, the coordinates of the marked points in each calibration board are uniformly converted to the same reference calibration board coordinate system, and the reference coordinates of different marked points in different calibration boards in the same coordinate system are obtained. Then, for different cameras to be calibrated, the second conversion relationship between the camera coordinate system of the camera to be calibrated and the calibration board coordinate system is respectively obtained by using the reference coordinates of the marked points in different calibration boards. Since the reference coordinates of the marked points in different calibration boards are coordinates in the same reference calibration board coordinate system, the above second conversion relationship can be understood as: the conversion relationship between the camera coordinate systems of different cameras to be calibrated and the same reference calibration board coordinate system. In view of the fact that the second conversion relationships of different cameras to be calibrated are established based on the same reference calibration board, the target conversion relationship between the camera coordinate systems of any two cameras to be calibrated can be directly calculated according to the above second conversion relationship. Thus, it can be seen that by applying the scheme provided by the above embodiments, the conversion relationship between the camera coordinate systems of multiple cameras can be calibrated.

[0202] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0203] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0204] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the apparatus embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content.

[0205] The above are only the preferred embodiments of the present application and are not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included within the protection scope of the present application.

Claims

1. A camera calibration method, characterized in that, The method includes: For at least two calibration plates among a plurality of calibration plates, obtaining a reference image collected by a reference camera and containing the at least two calibration plates, using the reference image to obtain a conversion relationship between a reference camera coordinate system of the reference camera and a calibration plate coordinate system of each calibration plate, and using the obtained conversion relationship to calculate a first conversion relationship between the calibration plate coordinate systems of the at least two calibration plates, wherein the plurality of calibration plates are respectively located in image acquisition regions of a plurality of cameras to be calibrated; According to the calculated first conversion relationship, determining a reference conversion relationship between the calibration plate coordinate systems of each calibration plate and a reference calibration plate coordinate system of a reference calibration plate, and using the reference conversion relationship to calculate reference coordinates of each calibration point in each calibration plate in the reference calibration plate coordinate system, wherein the reference calibration plate is any one of the plurality of calibration plates; For each camera to be calibrated, obtaining a target image of a target calibration plate collected by the camera to be calibrated, and using the target image and the reference coordinates of each calibration point in the target calibration plate to obtain a second conversion relationship between a camera coordinate system of the camera to be calibrated and the reference calibration plate coordinate system, wherein the target calibration plate is: the calibration plate corresponding to the camera to be calibrated, and the calibration plate corresponding to each camera to be calibrated is: the calibration plate located in the image acquisition region of the camera to be calibrated; For any two cameras to be calibrated, using the obtained second conversion relationships of the two cameras to be calibrated to calculate a target conversion relationship between the camera coordinate systems of the two cameras to be calibrated; The step of, for at least two calibration plates among a plurality of calibration plates, obtaining a reference image collected by a reference camera and containing the at least two calibration plates, using the reference image to obtain a conversion relationship between a reference camera coordinate system of the reference camera and a calibration plate coordinate system of each calibration plate, and using the obtained conversion relationship to calculate a first conversion relationship between the calibration plate coordinate systems of the at least two calibration plates, includes: For a first calibration plate and a second calibration plate among a plurality of calibration plates, obtaining a reference image collected by the reference camera and containing the first calibration plate and the second calibration plate, using the reference image to obtain a conversion relationship between the reference camera coordinate system of the reference camera and a first calibration plate coordinate system of the first calibration plate, and a conversion relationship between the reference camera coordinate system and a second calibration plate coordinate system of the second calibration plate, and using the obtained conversion relationships to calculate a first conversion relationship between the second calibration plate coordinate system and the first calibration plate coordinate system; Taking the second calibration plate as the first calibration plate and any calibration plate that has not obtained the first conversion relationship as the second calibration plate, and returning to the step of, for the first calibration plate and the second calibration plate among a plurality of calibration plates, obtaining a reference image collected by the reference camera and containing the first calibration plate and the second calibration plate, until all calibration plates are traversed; The step of, according to the calculated first conversion relationship, determining a reference conversion relationship between the calibration plate coordinate systems of each calibration plate and a reference calibration plate coordinate system of a reference calibration plate, includes: Obtain a reference image captured by the reference camera and containing a reference calibration board and a third calibration board. Using the reference image, obtain the conversion relationship between the reference camera coordinate system and the reference calibration board coordinate system of the reference calibration board, and the conversion relationship between the reference camera coordinate system and the third calibration board coordinate system of the third calibration board. Calculate the auxiliary conversion relationship between the third calibration board coordinate system and the reference calibration board coordinate system using the obtained conversion relationships, where the third calibration board is the finally determined second calibration board, and the reference calibration board is the initial first calibration board; Use the auxiliary conversion relationship to correct each first conversion relationship; Use the corrected first conversion relationships to determine the reference conversion relationships between the calibration board coordinate systems of each calibration board and the reference calibration board coordinate system; For any two cameras to be calibrated, calculate the target conversion relationship between the camera coordinate systems of the two cameras to be calibrated using the obtained second conversion relationships of the two cameras to be calibrated, including: For the i-th camera to be calibrated and the j-th camera to be calibrated, calculate the target transformation relationship between the camera coordinate systems of the i-th camera to be calibrated relative to the j-th camera to be calibrated according to the following formula Among them, the represents the second conversion relationship of the j-th camera to be calibrated, and the represents the second conversion relationship of the i-th camera to be calibrated.

2. The method according to claim 1, wherein Each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards; The using the auxiliary conversion relationship to correct each first conversion relationship includes: Taking the principle of reducing loss, use the auxiliary conversion relationship to correct each first conversion relationship, where the calculation method of the loss is: The objectionF represents the loss, the represents the auxiliary conversion relationship, the represents: the first conversion relationship between the calibration board coordinate system of the calibration board corresponding to the i-th camera to be calibrated and the calibration board coordinate system of the calibration board corresponding to the (i - 1)-th camera to be calibrated, where N represents the number of all cameras to be calibrated, and i represents the calibration board corresponding to the i-th camera to be calibrated, the represents the mapping for transforming to the corresponding Lie algebra, the represents the mapping for transforming to the corresponding Lie algebra.

3. The method according to claim 1, wherein For any two cameras to be calibrated, calculate the target conversion relationship between the camera coordinate systems of the two cameras to be calibrated using the obtained second conversion relationships of the two cameras to be calibrated, including: Optimize the second conversion relationships of each camera to be calibrated; For any two cameras to be calibrated, calculate the target conversion relationship between the camera coordinate systems of the two cameras to be calibrated using the optimized second conversion relationships of the two cameras to be calibrated.

4. The method according to claim 3, wherein Each camera to be calibrated corresponds to a calibration board, and different cameras to be calibrated correspond to different calibration boards; The optimizing the second conversion relationships of each camera to be calibrated includes: Optimize the second conversion relationships of each camera to be calibrated according to the following formula: Among them, the represents the second conversion relationship of the j-th camera to be calibrated, the S represents the optimized second conversion relationship of the j-th camera to be calibrated, the N represents the number of cameras to be calibrated, the j represents the j-th camera to be calibrated, the M represents the number of calibration points in each calibration board, the k represents the k-th calibration point, the S represents the number of images of the target images collected by each camera to be calibrated, the s represents the s-th target image collected, and the x iks represents the image coordinates of the k-th calibration point in the calibration board corresponding to the i-th camera to be calibrated in the s-th target image, and the X jk represents the reference coordinates of the k-th calibration point in the calibration board corresponding to the j-th camera to be calibrated.

5. A camera calibration device, characterized in that, The device includes: A first relationship obtaining module, configured to, for at least two calibration boards among a plurality of calibration boards, obtain a reference image captured by a reference camera and containing the at least two calibration boards. Using the reference image, obtain the conversion relationship between the reference camera coordinate system of the reference camera and the calibration board coordinate system of each calibration board, and calculate the first conversion relationship between the calibration board coordinate systems of the at least two calibration boards using the obtained conversion relationships, where the plurality of calibration boards are respectively located in the image acquisition areas of a plurality of cameras to be calibrated; A reference coordinate obtaining module, configured to determine the reference conversion relationship between the calibration board coordinate systems of each calibration board and the reference calibration board coordinate system of the reference calibration board according to the calculated first conversion relationship, and calculate the reference coordinates of each calibration point in each calibration board in the reference calibration board coordinate system using the reference conversion relationship, where the reference calibration board is any one of the plurality of calibration boards; A second relationship obtaining module, configured to, for each camera to be calibrated, obtain a target image of a target calibration board collected by the camera to be calibrated, and use the target image and the reference coordinates of each calibration point in the target calibration board to obtain a second conversion relationship between the camera coordinate system of the camera to be calibrated and the reference calibration board coordinate system, where the target calibration board is: the calibration board corresponding to the camera to be calibrated, and the calibration board corresponding to each camera to be calibrated is: the calibration board located within the image acquisition area of the camera to be calibrated; A target relationship obtaining module, configured to, for any two cameras to be calibrated, calculate a target conversion relationship between the camera coordinate systems of the two cameras to be calibrated by using the obtained second conversion relationships of the two cameras to be calibrated; The first relationship obtaining module includes: A first relationship obtaining unit, configured to, for a first calibration board and a second calibration board among a plurality of calibration boards, obtain a reference image collected by the reference camera and including the first calibration board and the second calibration board, and use the reference image to obtain a conversion relationship between the reference camera coordinate system of the reference camera and the first calibration board coordinate system of the first calibration board, and a conversion relationship between the reference camera coordinate system and the second calibration board coordinate system of the second calibration board, and calculate a first conversion relationship between the second calibration board coordinate system and the first calibration board coordinate system by using the obtained conversion relationships; A calibration board traversing unit, configured to use the second calibration board as the first calibration board and any calibration board for which the first conversion relationship has not been obtained as the second calibration board, and return to the step of obtaining a reference image collected by the reference camera and including the first calibration board and the second calibration board for the first calibration board and the second calibration board among the plurality of calibration boards, until all calibration boards are traversed; The reference coordinate obtaining module includes: An auxiliary relationship obtaining unit, configured to obtain a reference image collected by the reference camera and including a reference calibration board and a third calibration board, and use the reference image to obtain a conversion relationship between the reference camera coordinate system of the reference camera and the reference calibration board coordinate system of the reference calibration board, and a conversion relationship between the reference camera coordinate system and the third calibration board coordinate system of the third calibration board, and calculate an auxiliary conversion relationship between the third calibration board coordinate system and the reference calibration board coordinate system by using the obtained conversion relationships, where the third calibration board is: the second calibration board finally determined, and the reference calibration board is the initial first calibration board; A relationship correcting unit, configured to correct each first conversion relationship by using the auxiliary conversion relationship; A reference relationship obtaining unit, configured to determine a reference conversion relationship between the calibration board coordinate systems of each calibration board and the reference calibration board coordinate system by using the corrected first conversion relationships; A reference coordinate obtaining unit, configured to calculate the reference coordinates of each calibration point in each calibration board in the reference calibration board coordinate system by using the reference conversion relationship; The target relationship obtaining module is specifically configured to: For the i-th camera to be calibrated and the j-th camera to be calibrated, calculate the target transformation relationship between the camera coordinate systems of the i-th camera to be calibrated and the j-th camera to be calibrated according to the following formula Among them, the represents the second conversion relationship of the j-th camera to be calibrated, and the represents the second conversion relationship of the i-th camera to be calibrated.

6. An electronic device, characterized in that, Include a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory communicate with each other through the communication bus; A memory for storing a computer program; A processor, when executing the program stored in the memory, implements the method steps described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the method steps described in any one of claims 1-4 are implemented.

Images