Multi-camera joint calibration method, device, terminal equipment and readable storage medium
By acquiring and optimizing the calibration images of multi-camera, the problems of high constraints, low accuracy and insufficient robustness in the prior art are solved, and efficient and robust multi-camera calibration is achieved to adapt to complex operating environments.
Patent Information
- Application Number
- CN202011273007.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-13
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-11-13
AI Technical Summary
The existing multi-camera calibration technology achieves high constraints, low calibration accuracy and insufficient robustness, making it unable to adapt to complex operating environments.
By acquiring the calibration images of multiple cameras, determining the initial relative attitude data, calculating and optimizing the reprojection error, iteratively computing the relative attitude data between cameras, realizing automatic calibration and adapting to complex environments.
Improve calibration accuracy, efficiency and robustness between multiple cameras, and reduce reprojection errors.
Smart Images

Figure CN112330752B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and in particular, relates to a multi-camera joint calibration method, apparatus, terminal device and readable storage medium. Background Art
[0002] In recent years, robots have been widely used in various industries. In order to cope with complex and changeable tasks, multiple robots are often used to work together. In the process of multi-robot collaborative work, camera calibration technology is often used to determine the relative posture relationship between multiple robots to improve the efficiency of multi-robot collaborative work.
[0003] The multi-camera calibration technology currently used is mainly based on the correspondence between the two images generated during the camera movement. This method is highly flexible, but its implementation constraints are high, the calibration accuracy is low, and the robustness is insufficient, and it cannot adapt to complex working environments. Summary of the invention
[0004] The embodiments of the present application provide a multi-camera joint calibration method, apparatus, terminal device and readable storage medium, which can solve the problems of the current camera calibration technology having high implementation constraints, low calibration accuracy and insufficient robustness, and being unable to adapt to complex operating environments.
[0005] In a first aspect, an embodiment of the present application provides a multi-camera joint calibration method, including:
[0006] Acquire a first calibration image of the first camera and a second calibration image of the second camera respectively; wherein the calibration image is an image obtained by the camera shooting a calibration plate located at the same position;
[0007] Determining initial relative posture data between the first camera and the second camera according to the first calibration image and the second calibration image;
[0008] Calculate and obtain an initial reprojection error between the first camera and the second camera according to the initial relative posture data;
[0009] Performing iterative optimization calculation on the initial reprojection error to determine a second reprojection error;
[0010] Relative posture data between the first camera and the second camera is determined according to the second reprojection error.
[0011] In a second aspect, an embodiment of the present application provides a multi-camera joint calibration device, including:
[0012] An acquisition module, used to respectively acquire a first calibration image of the first camera and a second calibration image of the second camera; wherein the calibration image is an image obtained by the camera shooting a calibration plate located at the same position;
[0013] A first determination module, configured to determine initial relative posture data between the first camera and the second camera according to the first calibration image and the second calibration image;
[0014] A calculation module, configured to calculate an initial reprojection error between the first camera and the second camera according to the initial relative posture data;
[0015] An error optimization module, used for iteratively optimizing the initial reprojection error to determine a second reprojection error;
[0016] The second determination module is used to determine the relative posture data between the first camera and the second camera according to the second reprojection error.
[0017] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-camera joint calibration method as described in any one of the first aspects above is implemented.
[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multi-camera joint calibration method as described in any one of the first aspects above is implemented.
[0019] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the multi-camera joint calibration method described in any one of the first aspects above.
[0020] Automated calibration is achieved by automatically acquiring the camera's calibration images. It can adapt to complex environments, perform iterative error optimization on the reprojection error, obtain the optimized relative posture data between cameras, reduce the reprojection error between multiple cameras, and improve the calibration accuracy and efficiency between multiple cameras, as well as the robustness of the multi-camera joint calibration technology.
[0021] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0023] Figure 1 is a structural schematic diagram of a multi-camera joint calibration system provided in an embodiment of the present application;
[0024] Figure 2 is a flowchart of a multi-camera joint calibration method provided in an embodiment of the present application;
[0025] Figure 3 is a schematic diagram of a calibration plate provided in an embodiment of the present application;
[0026] Figure 4 is a schematic diagram of an application scenario in which there is an overlapping field of view between a first camera and a second camera provided in an embodiment of the present application;
[0027] Figure 5 is a schematic diagram of an application scenario in which there is no overlapping field of view between the first camera and the second camera provided in an embodiment of the present application;
[0028] Figure 6 is a structural schematic diagram of a multi-camera joint calibration device provided in an embodiment of the present application;
[0029] Figure 7 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0030] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0031] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0032] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0033] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0034] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0035] The multi-camera joint calibration method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, and laptop computers. The embodiment of the present application does not impose any restrictions on the specific type of terminal devices.
[0036] In recent years, although the technology of determining the relative posture relationship between multiple robots through multi-camera calibration technology has been widely promoted, the existing multi-camera calibration technology has high constraints, low calibration accuracy and insufficient robustness, and cannot adapt to complex working environments. To solve this problem, the present application proposes a multi-camera joint calibration method, a multi-camera joint calibration device, a terminal device and a readable storage medium, which can realize automated multi-camera joint calibration technology, can adapt to complex working environments, reduce the reprojection error between multiple cameras, improve the calibration accuracy and calibration efficiency between multiple cameras, and the robustness of the multi-camera joint calibration technology.
[0037] To implement the technical solution proposed in this application, a multi-camera joint calibration system can be constructed first. Figure 1 The camera joint calibration system consists of more than one robot J( Figure 1 Only three are shown in the figure, among which each robot J needs to be set with a camera C), an intelligent mobile car (the intelligent mobile car is fixed with a three-dimensional calibration plate, which is communicated with the intelligent gimbal with variable angles) and a terminal device, and the robot, camera, intelligent mobile car and intelligent gimbal are all communicated with the terminal device.
[0038] Among them, the robot is a robot that carries a camera and has a positioning service demand in order to achieve collaborative work with other robots; the intelligent gimbal is a gimbal device that rotates according to the rotation instruction of the terminal device and then rotates the three-dimensional calibration plate; the intelligent mobile car is an intelligent car that carries a three-dimensional calibration plate and moves according to the movement instruction of the terminal device; the terminal device is a terminal device that controls the rotation of the intelligent gimbal, controls the movement of the intelligent mobile car, controls the camera carried by the robot to take pictures and obtains the calibration image taken by the camera, performs joint calibration processing according to the calibration image, determines the relative posture between the cameras, and then determines the relative posture relationship between the robots corresponding to the cameras.
[0039] In the process of automated multi-camera joint calibration, the terminal device can send movement instructions to the intelligent mobile car to control the intelligent mobile car to reach the specified position, and send rotation instructions to the intelligent gimbal to control the intelligent gimbal to rotate so that the stereo calibration plate faces the camera carried by the robot. At the same time, it sends shooting instructions to control the camera carried by the robot to shoot the stereo calibration plate, obtain the calibration images sent by all cameras, associate and store the camera calibration images, camera numbers and the location information of the stereo calibration plate, and then perform joint calibration processing on the calibration images of the associated cameras to determine the relative posture relationship between multiple cameras to obtain the relative posture relationship between the robots corresponding to the cameras.
[0040] In order to illustrate the technical solution proposed in this application, a specific embodiment is provided below for illustration.
[0041] Figure 2 A schematic flowchart of the multi-camera joint calibration method provided in the present application is shown. As an example but not a limitation, the method can be applied to the above-mentioned laptop computer.
[0042] S101, respectively obtaining a first calibration image of a first camera and a second calibration image of a second camera; wherein the calibration image is an image obtained by taking a calibration plate located at the same position by a camera.
[0043] In a specific application, the intelligent mobile car is first controlled to move to the destination point, and the intelligent gimbal is controlled to rotate so that the calibration plate faces the camera, and then the first calibration image sent by the first camera and the second calibration image sent by the second camera are obtained, wherein the calibration image is an image obtained by shooting the calibration plate at the same position by the first camera and the second camera. The calibration plate is a five-sided three-dimensional checkerboard calibration plate (such as Figure 3). When the first camera and the second camera have overlapping fields of view, the destination of the smart mobile car is located in the overlapping fields of view of the two cameras; when the first camera and the second camera do not have overlapping fields of view, the destination of the smart mobile car is in the overlapping fields of view of the third camera and the first camera and in the overlapping fields of view of the third camera and the second camera. The third camera is any camera other than the first camera and the second camera that has overlapping fields of view with the first camera and the second camera.
[0044] like Figure 3 As shown, a schematic diagram of a calibration plate is provided;
[0045] Figure 3 Each side of the stereo calibration plate is a 7×10 checkerboard calibration plate. The angles between calibration plates 2, 3, 4, and 5 and calibration plate 1 are 45°, so as to detect more corner point information in the calibration images taken by cameras at different orientations. The corner point is the extreme point in the calibration image.
[0046] S102: Determine initial relative posture data between the first camera and the second camera according to the first calibration image and the second calibration image.
[0047] In a specific application, the parameters of the first camera are calculated based on the calibration plate data in the first calibration image of the first camera; the parameters of the second camera are calculated based on the calibration plate data in the second calibration image of the second camera, and then the initial relative posture data between the first camera and the second camera are calculated based on the parameters of the first camera and the second camera. The camera parameters include but are not limited to the camera's intrinsic parameter matrix K and the single camera rotation and translation matrix (R, t); the calibration plate data is the position data of the corner points detected in the calibration image.
[0048] S103: Calculate an initial reprojection error between the first camera and the second camera according to the initial relative posture data.
[0049] In a specific application, corner point detection is performed on the first calibration image and the second calibration image, calibration plate data in the first calibration image and the second calibration image is determined, common corner points of the first camera and the second camera are determined according to the calibration plate data, and the two-dimensional coordinates of the common corner points in the first calibration image (i.e., the target points) are obtained. According to the initial relative posture data between the first camera and the second camera, the two-dimensional coordinates of the target point are transformed to obtain the two-dimensional coordinates of the estimated point, and then the initial reprojection error between the first camera and the second camera is calculated according to the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point.
[0050] S104: performing iterative optimization calculation on the initial reprojection error to determine a second reprojection error.
[0051] In a specific application, a reprojection error optimization model is established according to the position coordinates of the common corner point between the first camera and the second camera, and the reprojection error optimization model is iteratively optimized to determine the second reprojection error. The second reprojection error refers to the minimum reprojection error obtained by iterative optimization calculation that meets the preset conditions.
[0052] S105: Determine relative posture data between the first camera and the second camera according to the second reprojection error.
[0053] In a specific application, the relative posture data between the first camera and the second camera after iterative optimization is determined according to the second reprojection error, and then the relative posture relationship between the first robot corresponding to the first camera and the second robot corresponding to the second camera is determined through the relative posture data between the first camera and the second camera.
[0054] It is understandable that after the relative posture relationship between the first robot and the second robot is determined, the relative posture relationship data should be sent to the first robot and the second robot to facilitate the robot to determine the position relationship between itself and the other robot.
[0055] In a specific application, the reprojection error optimization model may be iteratively optimized by a preset algorithm to determine the second reprojection error, wherein the preset algorithm includes but is not limited to the Levenberg-Marquardt algorithm.
[0056] Where: The reprojection error f(x) is optimized by K iterations through the Levenberg-Marquardt algorithm as shown in formulas (1) and (2):
[0057]
[0058] ||DΔx k ||2≤μ (2);
[0059] Among them, x k represents the initial relative posture data after K optimizations, x k+1 represents the initial relative posture data after K+1 optimizations, Δx k Indicates the pair x obtained during the k+1th optimization process k The correction amount, f(x k ) represents the two-dimensional coordinates of the estimated point after the Kth optimization; J[x k ] means f(x k ) is the first-order derivative with respect to x, and D is the coefficient matrix.
[0060] In a specific application, the implementation steps of iteratively optimizing the reprojection error f(x) by the Levenberg-Marquardt algorithm are as follows:
[0061] Given an initial value x0, the initial value x0 is the initial relative posture data R between the first camera and the second camera 12 ; Set the initial optimization radius μ; wherein, the initial optimization radius μ can be set specifically according to actual conditions.
[0062] After K iterations of optimizing the reprojection error f(x), we calculate The value of ; where ρ is an indicator set in the Levenberg-Marquardt algorithm to characterize the quality of the approximate second-order Taylor expansion used in the Gauss-Newton method. T Δx k is the value of the actual function drop, the denominator f(x+Δx k )-f(x) is the value of the approximate model descent.
[0063] Check the size of ρ. If the calculated ρ value is small, the optimization radius μ should be reduced (in this embodiment, set When , the optimization radius μ is reduced, and μ = 0.5μ); if the calculated ρ value is large, the optimization radius μ should be enlarged (in this embodiment, When , enlarge the optimization radius μ and set μ=2μ).
[0064] When it is detected that ρ is greater than the preset threshold, it is determined that the approximation adopted in this iteration process is feasible, and x k+1 =x k +Δx k ; At this time, the algorithm can be judged whether it has converged by comparing the reprojection error after iterative optimization with the preset reprojection error threshold; if the reprojection error after iterative optimization is less than or equal to the preset reprojection error threshold, the iterative optimization is considered to be completed, and the reprojection error after iterative optimization is used as the second reprojection error. At this time, x k+1 That is, the relative posture data between the first camera and the second camera. If the reprojection error after iterative optimization is greater than the preset reprojection error threshold, the k+2th iterative optimization is continued.
[0065] According to the relative posture data x between the first camera and the second camera obtained after the above iterative optimization k+1 The reprojection error between the first camera and the second camera is calculated again, and it can be detected that the reprojection error at this time is much smaller than the initial reprojection error (the reprojection error at this time is about 10% of the initial reprojection error), thereby improving the calibration accuracy between multiple cameras.
[0066] In one embodiment, the step S102 includes:
[0067] S1021, obtaining calibration plate data of the first calibration image and the second calibration image; wherein the calibration plate data is position data of corner points in the calibration image;
[0068] S1022: Calculate and obtain parameters of the first camera and the second camera respectively according to the calibration plate data of the first calibration image and the second calibration image;
[0069] S1023: Calculate and obtain initial relative posture data between the first camera and the second camera according to parameters of the first camera and the second camera.
[0070] In a specific application, the calibration plate data of the first calibration image and the calibration plate data of the second calibration image are obtained, the parameters of the first camera are calculated based on the calibration plate data of the first calibration image, the parameters of the second camera are calculated based on the calibration plate data of the second calibration image, and the initial relative posture data between the first camera and the second camera are calculated based on the parameters of the first camera and the parameters of the second camera. Among them, the parameters of the camera include but are not limited to the intrinsic parameter matrix K of the camera and the single camera rotation and translation matrix (R, t). The single camera rotation and translation matrix is specifically the rotation and translation matrix between the camera and the calibration plate in the overlapping field of view. The calibration plate data is the position data of the corner points identified in the calibration image. The position data of the corner points are the position coordinates of the corner points in the calibration image.
[0071] In this embodiment, the parameters of the camera can be calculated using the Zhang Zhengyou calibration method.
[0072] In one embodiment, the step S1023 includes:
[0073] If there is an overlapping field of view between the first camera and the second camera, initial relative posture data between the first camera and the second camera is calculated according to parameters of the first camera and the second camera.
[0074] In a specific application, if there is an overlapping field of view between the first camera and the second camera, the initial relative posture data between the first camera and the second camera can be calculated according to the parameters of the first camera and the second camera.
[0075] like Figure 4 As shown, a schematic diagram of an application scenario in which a first camera and a second camera have overlapping fields of view is provided;
[0076] Figure 4In the example, there is an overlapping field of view between the first camera and the second camera. First, the intelligent mobile car should be controlled to move into the overlapping field of view between the first camera and the second camera, and the calibration plate should be rotated to face the first camera and the second camera by rotating the intelligent gimbal. By acquiring the first calibration image of the first camera and the second calibration image of the second camera, the intrinsic parameter matrix of the first camera, the intrinsic parameter matrix of the second camera, the single camera rotation matrix R1 of the first camera relative to the calibration plate in the overlapping field of view, and the single camera rotation matrix R2 of the second camera relative to the calibration plate in the overlapping field of view are calculated respectively. The initial relative posture data R between the first camera and the second camera is calculated according to formula (3): 12 .
[0077] R 12 =R1 -1 R2 formula (3).
[0078] In one embodiment, the step S1023 further includes:
[0079] If there is no overlapping field of view between the first camera and the second camera, first relative posture data between the first camera and a third camera, and second relative posture data between the second camera and the third camera are calculated and obtained; wherein the third camera is any camera other than the first camera and the second camera, and has an overlapping field of view with the first camera and the second camera respectively;
[0080] Initial relative posture data between the first camera and the second camera is obtained by calculation according to the first relative posture data and the second relative posture data.
[0081] In a specific application, if it is detected that there is no overlapping field of view between the first camera and the second camera, a third calibration image of the third camera is obtained. According to the calibration plate data of the third calibration image and the calibration plate data of the first calibration image, the first relative posture data between the first camera and the third camera is calculated; according to the calibration plate data of the third calibration image and the calibration plate data of the second calibration image, the second relative posture data between the second camera and the third camera is calculated; then, according to the first relative posture data and the second relative posture data, the initial relative posture data between the first camera and the second camera is calculated.
[0082] like Figure 5 As shown, a schematic diagram of an application scenario is provided in which there is no overlapping field of view between a first camera and a second camera.
[0083] Figure 5In the example, there is no overlapping field of view between the first camera and the second camera, so a third camera should be determined. The intelligent mobile car is controlled to move into the overlapping field of view between the first camera and the third camera, and the calibration plate is made to face the first camera and the third camera by rotating the intelligent gimbal. The first calibration image of the first camera and the third calibration image of the third camera are obtained by calculation, and the intrinsic parameter matrix of the first camera, the intrinsic parameter matrix of the third camera, the single camera rotation matrix R1 of the first camera relative to the calibration plate in the overlapping field of view, and the single camera rotation matrix R3 of the third camera relative to the calibration plate in the overlapping field of view are obtained, and then the first relative posture data R between the first camera and the third camera is calculated. 13 Then the intelligent mobile car is controlled to move to the overlapping field of view between the second camera and the third camera, and the calibration plate is rotated to face the second camera and the third camera, and the second relative posture data R between the second camera and the third camera is obtained according to the above calculation method. 23 Finally, according to the first relative posture data R between the first camera and the third camera, formula (4) is used 13 and second relative posture data R between the second camera and the third camera 23 , calculate and obtain the initial relative posture data R between the first camera and the second camera 12 , as follows:
[0084] R 12 =R 13 -1 ·R 23 Formula (4).
[0085] The initial relative pose data between cameras with overlapping fields of view are used to calculate the initial relative pose data between cameras without overlapping fields of view. This can be applied to working environments where some cameras do not have overlapping fields of view, thereby improving the robustness of the multi-camera joint calibration method.
[0086] In one embodiment, the step S103 includes:
[0087] Determining a common corner point of the first camera and the second camera;
[0088] Taking the common corner point in the first calibration image as a target point, acquiring the three-dimensional coordinates and the two-dimensional coordinates of the target point;
[0089] According to the initial relative posture data between the first camera and the second camera, the three-dimensional coordinates of the target point are transformed to obtain the two-dimensional coordinates of the estimated point; wherein the estimated point is the common corner point in the second calibration image;
[0090] An initial reprojection error between the first camera and the second camera is calculated based on the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point.
[0091] In a specific application, the common corner points of the first camera and the second camera are determined according to the calibration plate data of the first calibration image of the first camera and the calibration plate data in the second calibration image of the second camera. The common corner points in the first calibration image are taken as the target points, the three-dimensional coordinates of the target points are obtained, and the first camera is calibrated by a single camera to obtain the two-dimensional coordinates of the target points; according to the intrinsic parameter matrix of the first camera, the intrinsic parameter matrix of the second camera and the initial relative posture data between the first camera and the second camera, the three-dimensional coordinates of the target points are transformed to obtain the two-dimensional coordinates of the estimated points (as shown in formula (5)). According to the two-dimensional coordinates of the target points and the two-dimensional coordinates of the estimated points, the initial reprojection error between the first camera and the second camera is calculated (as shown in formula (6)). The estimated points are the common corner points in the second calibration image.
[0092] In a specific application, the common corner point is a corner point shared by the first calibration image of the first camera and the second calibration image of the second camera. For example, if the second calibration image of the first camera has corner points of the first surface and corner points of the fourth surface, and the second calibration image of the second camera has corner points of the first surface and corner points of the fifth surface, then the corner point of the first surface is set as the common corner point between the first camera and the second camera.
[0093]
[0094] error=||P 2D -P 3D_2D ||2 (6);
[0095] In formula (5), P 3D represents the three-dimensional coordinates of the target point in the first calibration image;
[0096] P 2D represents the two-dimensional coordinates of the target point obtained by performing single-camera calibration on the first camera;
[0097] P 3D_2D Represents the two-dimensional coordinates of the estimated point.
[0098] In one embodiment, the step S104 includes:
[0099] Perform global modeling based on the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point to obtain a reprojection error optimization model;
[0100] The reprojection error optimization model is iteratively calculated to determine the second reprojection error that meets the preset conditions.
[0101] In a specific application, global modeling is performed based on the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point to obtain an optimization model of the reprojection error (as shown in formula (7)). The reprojection error optimization model is iteratively optimized and calculated using a preset algorithm to determine the second reprojection error that meets the preset conditions.
[0102] Among them, the preset condition can be specifically set according to the actual situation. In this embodiment, the preset condition is set to obtain a reprojection error that is less than or equal to a preset reprojection error threshold. That is, when it is determined that the reprojection error after iterative optimization is less than or equal to the preset reprojection error threshold, the iterative optimization of the judgment algorithm is completed, and the reprojection error that is less than or equal to the preset reprojection error threshold is the second reprojection error. The preset reprojection error threshold can be specifically set according to actual needs. For example, the preset reprojection error threshold is set to 15% of the initial reprojection error.
[0103] The established reprojection error optimization model is as shown in formula (7):
[0104]
[0105] In formula (7), f(x) represents the estimated point P 3D_2D The two-dimensional coordinates, P 3D_2D =K[R, t]P 3D ; y represents the target point P obtained by performing single-camera calibration on the first camera 2D The value of f(x) is set according to the image data format of the calibration image. For example, if the resolution of the current calibration image is 600×800, the corresponding 0≤f(x)≤800.
[0106] Automated calibration is achieved by automatically acquiring the camera's calibration images. It can adapt to complex environments, perform iterative error optimization on the reprojection error, obtain the optimized relative posture data between cameras, reduce the reprojection error between multiple cameras, and improve the calibration accuracy and efficiency between multiple cameras and the robustness of the multi-camera joint calibration technology.
[0107] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0108] Corresponding to the multi-camera joint calibration method described in the above embodiment, Figure 6 A structural block diagram of a multi-camera joint calibration device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0109] Reference Figure 6 , the multi-camera joint calibration device 100 comprises:
[0110] An acquisition module 101 is used to acquire a first calibration image of a first camera and a second calibration image of a second camera respectively; wherein the calibration image is an image obtained by the camera shooting a calibration plate located at the same position;
[0111] A first determination module 102, configured to determine initial relative posture data between the first camera and the second camera according to the first calibration image and the second calibration image;
[0112] A calculation module 103, configured to calculate an initial reprojection error between the first camera and the second camera according to the initial relative posture data;
[0113] The error optimization module 104 is used to perform iterative optimization calculation on the initial reprojection error to determine a second reprojection error;
[0114] The second determination module 105 is used to determine the relative posture data between the first camera and the second camera according to the second reprojection error.
[0115] In one embodiment, the first determining module includes:
[0116] A first acquisition unit, configured to acquire calibration plate data of the first calibration image and the second calibration image; wherein the calibration plate data is position data of corner points in the calibration image;
[0117] A first calculation unit, configured to calculate parameters of the first camera and the second camera respectively according to the calibration plate data of the first calibration image and the second calibration image;
[0118] The second calculation unit is used to calculate and obtain initial relative posture data between the first camera and the second camera according to parameters of the first camera and the second camera.
[0119] In one embodiment, the second computing unit includes:
[0120] The first calculation subunit is configured to calculate and obtain initial relative posture data between the first camera and the second camera according to parameters of the first camera and the second camera if there is an overlapping field of view between the first camera and the second camera.
[0121] In one embodiment, the second computing unit further includes:
[0122] a second calculation subunit, configured to calculate and obtain first relative posture data between the first camera and a third camera, and second relative posture data between the second camera and the third camera, if there is no overlapping field of view between the first camera and the second camera; wherein the third camera is any camera other than the first camera and the second camera, and has an overlapping field of view with the first camera and the second camera respectively;
[0123] The third calculation subunit is used to calculate and obtain initial relative posture data between the first camera and the second camera according to the first relative posture data and the second relative posture data.
[0124] In one embodiment, the calculation module 103 includes:
[0125] A first determining unit, configured to determine a common corner point of the first camera and the second camera;
[0126] A second acquisition unit, configured to take the common corner point in the first calibration image as a target point and acquire the three-dimensional coordinates and the two-dimensional coordinates of the target point;
[0127] a conversion unit, configured to convert the three-dimensional coordinates of the target point according to the initial relative posture data between the first camera and the second camera to obtain the two-dimensional coordinates of an estimated point; wherein the estimated point is the common corner point in the second calibration image;
[0128] The third calculation unit is used to calculate and obtain an initial reprojection error between the first camera and the second camera according to the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point.
[0129] In one embodiment, the error optimization module 104 includes:
[0130] A modeling unit, used for performing global modeling according to the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point to obtain a reprojection error optimization model;
[0131] The second determination unit is used to iteratively calculate the reprojection error optimization model to determine a second reprojection error that meets a preset condition.
[0132] Automated calibration is achieved by automatically acquiring the camera's calibration images. It can adapt to complex environments, perform iterative error optimization on the reprojection error, obtain the optimized relative posture data between cameras, reduce the reprojection error between multiple cameras, and improve the calibration accuracy and efficiency between multiple cameras, as well as the robustness of the multi-camera joint calibration technology.
[0133] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0134] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Figure 7 As shown, the terminal device 7 of this embodiment includes: at least one processor 70 ( Figure 7 Only one is shown in the figure) a processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, and when the processor 70 executes the computer program 72, the steps in any of the above-mentioned embodiments of the multi-camera joint calibration method are implemented.
[0135] The terminal device 7 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.
[0136] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0137] In some embodiments, the memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. In other embodiments, the memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital, SD), a flash card (Flash Card), etc. equipped on the terminal device 7. Further, the memory 71 may also include both an internal storage unit of the terminal device 7 and an external storage device. The memory 71 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 71 may also be used to temporarily store data that has been output or is to be output.
[0138] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0139] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0140] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0142] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0143] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0144] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0146] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A multi-camera joint calibration method is applied to a multi-camera joint calibration system, wherein the multi-camera joint calibration system is composed of one or more robots, an intelligent mobile vehicle, and a terminal device, wherein: A camera needs to be set on each robot, and the intelligent mobile car is fixed with a three-dimensional calibration plate, which is connected to the intelligent pan-tilt with variable angles, and the robot, camera, intelligent mobile car, and intelligent pan-tilt are all connected to the terminal device. The robot is: a robot carrying a camera and having a positioning service demand to achieve collaborative operation with other robots; the intelligent pan-tilt is a pan-tilt device that rotates according to the rotation instruction of the terminal device and then rotates the three-dimensional calibration plate; the intelligent mobile car is an intelligent car that carries the three-dimensional calibration plate and moves according to the movement instruction of the terminal device; the terminal device controls the rotation of the intelligent pan-tilt, controls the movement of the intelligent mobile car, controls the camera carried by the robot to shoot and obtains the calibration image taken by the camera, performs joint calibration processing according to the calibration image, determines the relative posture between the cameras, and then determines the relative posture relationship between the robots corresponding to the cameras, and is characterized in that it includes: Respectively obtaining a first calibration image of the first camera and a second calibration image of the second camera, including: controlling the intelligent mobile car to move to the destination point, controlling the intelligent pan / tilt to rotate so that the calibration plate faces the camera, and then obtaining the first calibration image sent by the first camera and the second calibration image sent by the second camera, wherein the calibration image is an image obtained by shooting the calibration plate located at the same position by the first camera and the second camera; wherein the calibration plate is a five-sided three-dimensional chessboard calibration plate; each side of the three-dimensional calibration plate is a 7×10 chessboard calibration plate; wherein the angles between calibration plates No. 2, No. 3, No. 4, and No. 5 and calibration plate No. 1 are 45° respectively; Determining initial relative posture data between the first camera and the second camera according to the first calibration image and the second calibration image; Calculating an initial reprojection error between the first camera and the second camera according to the initial relative posture data includes: Determining a common corner point of the first camera and the second camera; Taking the common corner point in the first calibration image as a target point, acquiring the three-dimensional coordinates and the two-dimensional coordinates of the target point; According to the initial relative posture data between the first camera and the second camera, the three-dimensional coordinates of the target point are transformed to obtain the two-dimensional coordinates of the estimated point; wherein the estimated point is the common corner point in the second calibration image; Calculate and obtain an initial reprojection error between the first camera and the second camera according to the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point; Performing iterative optimization calculation on the initial reprojection error to determine a second reprojection error; Relative posture data between the first camera and the second camera is determined according to the second reprojection error.
2. The multi-camera joint calibration method according to claim 1, characterized in that: The determining, according to the first calibration image and the second calibration image, initial relative posture data between the first camera and the second camera comprises: Acquire calibration plate data of the first calibration image and the second calibration image; wherein the calibration plate data is position data of corner points in the calibration image; Calculate and obtain parameters of the first camera and the second camera respectively according to the calibration plate data of the first calibration image and the second calibration image; Initial relative posture data between the first camera and the second camera is calculated and obtained according to the parameters of the first camera and the second camera.
3. The multi-camera joint calibration method according to claim 2, characterized in that: The step of calculating and obtaining initial relative posture data between the first camera and the second camera according to the parameters of the first camera and the second camera includes: If there is an overlapping field of view between the first camera and the second camera, initial relative posture data between the first camera and the second camera is calculated according to parameters of the first camera and the second camera.
4. The multi-camera joint calibration method according to claim 2, characterized in that: The step of calculating and obtaining initial relative posture data between the first camera and the second camera according to the parameters of the first camera and the second camera further includes: If there is no overlapping field of view between the first camera and the second camera, first relative posture data between the first camera and a third camera, and second relative posture data between the second camera and the third camera are calculated and obtained; wherein the third camera is any camera other than the first camera and the second camera, and has an overlapping field of view with the first camera and the second camera respectively; Initial relative posture data between the first camera and the second camera is obtained by calculation according to the first relative posture data and the second relative posture data.
5. The multi-camera joint calibration method according to claim 1, characterized in that: The iterative optimization calculation of the initial reprojection error to determine the second reprojection error includes: Perform global modeling based on the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point to obtain a reprojection error optimization model; The reprojection error optimization model is iteratively calculated to determine the second reprojection error that meets the preset conditions.
6. A multi-camera joint calibration device is applied to a multi-camera joint calibration system, wherein the multi-camera joint calibration system is composed of one or more robots, an intelligent mobile vehicle, and a terminal device, wherein: A camera needs to be set on each robot, and the intelligent mobile car is fixed with a three-dimensional calibration plate, which is connected to the intelligent pan-tilt with variable angles, and the robot, camera, intelligent mobile car, and intelligent pan-tilt are all connected to the terminal device. The robot is: a robot carrying a camera and having a positioning service demand to achieve collaborative operation with other robots; the intelligent pan-tilt is a pan-tilt device that rotates according to the rotation instruction of the terminal device and then rotates the three-dimensional calibration plate; the intelligent mobile car is an intelligent car that carries the three-dimensional calibration plate and moves according to the movement instruction of the terminal device; the terminal device controls the rotation of the intelligent pan-tilt, controls the movement of the intelligent mobile car, controls the camera carried by the robot to shoot and obtains the calibration image taken by the camera, performs joint calibration processing according to the calibration image, determines the relative posture between the cameras, and then determines the relative posture relationship between the robots corresponding to the cameras, and is characterized in that it includes: An acquisition module, used to respectively acquire a first calibration image of the first camera and a second calibration image of the second camera, including: controlling the intelligent mobile vehicle to move to the destination point, controlling the intelligent pan / tilt to rotate so that the calibration plate faces the camera, and then acquiring the first calibration image sent by the first camera and the second calibration image sent by the second camera, wherein the calibration image is an image obtained by shooting the calibration plate at the same position by the first camera and the second camera; wherein the calibration plate is a five-sided three-dimensional checkerboard calibration plate; each side of the three-dimensional calibration plate is a 7×10 checkerboard calibration plate; wherein the angles between calibration plates No. 2, No. 3, No. 4, and No. 5 and calibration plate No. 1 are 45° respectively; a first determining module, configured to determine initial relative posture data between the first camera and the second camera according to the first calibration image and the second calibration image; A calculation module, used for calculating an initial reprojection error between the first camera and the second camera according to the initial relative posture data, comprising: Determining a common corner point of the first camera and the second camera; Taking the common corner point in the first calibration image as a target point, acquiring the three-dimensional coordinates and the two-dimensional coordinates of the target point; According to the initial relative posture data between the first camera and the second camera, the three-dimensional coordinates of the target point are transformed to obtain the two-dimensional coordinates of the estimated point; wherein the estimated point is the common corner point in the second calibration image; Calculate and obtain an initial reprojection error between the first camera and the second camera according to the two-dimensional coordinates of the target point and the two-dimensional coordinates of the estimated point; Performing iterative optimization calculation on the initial reprojection error to determine a second reprojection error; An error optimization module, used for iteratively optimizing the initial reprojection error to determine a second reprojection error; The second determination module is used to determine the relative posture data between the first camera and the second camera according to the second reprojection error.
7. The multi-camera joint calibration device according to claim 6, characterized in that: The first determining module includes: A first acquisition unit, configured to acquire calibration plate data of the first calibration image and the second calibration image; wherein the calibration plate data is position data of corner points in the calibration image; A first calculation unit, configured to calculate parameters of the first camera and the second camera respectively according to the calibration plate data of the first calibration image and the second calibration image; The second calculation unit is used to calculate and obtain initial relative posture data between the first camera and the second camera according to parameters of the first camera and the second camera.
8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.
9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
