A camera calibration method, device and electronic device

Abstract

Embodiments of the present application provide a camera calibration method, apparatus, and electronic device, including: obtaining at least two image data of a target array captured by a camera, a first target coordinate of a main target, a first conversion relationship, and a second target coordinate of key points of auxiliary targets; detecting target key points in each image data to obtain a first image coordinate of the main target key point and a second image coordinate of the auxiliary target key point in each image data; determining a second conversion relationship between the main target coordinate system and the lens coordinate system according to the second image coordinate and the first target coordinate; determining a third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data; and calibrating parameters of the camera. Embodiments of the present application can improve the efficiency of camera calibration. Especially for a camera with a large field of view, by constraining the pose relationship between targets, it can compensate for the loss of calibration accuracy caused by the small size of the targets in the field of view, provide robust constraints, and improve the robustness and accuracy of calibration.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and in particular, to a camera calibration method, device, and electronic device. Background Art

[0002] Camera calibration refers to determining the parameters of a camera geometric model by establishing the mutual relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image captured by the camera. For cameras with a large field of view, using a target of normal size usually cannot cover most of the field of view, which will affect the positioning accuracy of the target feature points and thus make it difficult to achieve accurate calibration. Therefore, in order to meet the calibration accuracy of the camera within the field of view, the traditional camera calibration process usually requires using a large target board, calibration tooling, etc. as three-dimensional spatial objects for calibration. Such target boards, calibration tooling, etc. are often large in size and difficult to guarantee the processing accuracy, and are not easily obtained. However, if an ordinary two-dimensional target (i.e., a planar target) is used, the calibration process may be too complex and it is difficult to guarantee the calibration accuracy, resulting in a low calibration efficiency of the camera. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a camera calibration method, device, and electronic device to solve at least one of the above problems. The specific technical solutions are as follows:

[0004] In a first aspect, this application provides a camera calibration method, including:

[0005] Obtaining at least two image data of a target array captured by a camera, and obtaining the first target coordinates of the main target key points of the main target in the main target coordinate system; wherein, the camera includes at least one lens, each of the image data is captured by the lens at different positions, the target array includes the main target and at least one auxiliary target, and the poses of the main target and the auxiliary target remain unchanged during the process of capturing the image data;

[0006] For each auxiliary target, obtaining the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system, and obtaining the second target coordinates of the auxiliary target key points of the auxiliary target in the auxiliary target coordinate system of the auxiliary target;

[0007] Performing target key point detection on each of the image data to respectively obtain the first image coordinates of the main target key points and the second image coordinates of the auxiliary target key points in each image data;

[0008] For each piece of image data, determine a second conversion relationship between the main target coordinate system and the lens coordinate system corresponding to the image data according to the first image coordinates in the image data and the first target coordinates; wherein, the lens coordinate system corresponding to the image data is the lens coordinate system when the camera captures the image data.

[0009] Determine a third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data according to the second conversion relationship and the first conversion relationship.

[0010] Calibrate the parameters of the camera according to the third conversion relationship corresponding to each piece of image data, the second image coordinates in each piece of image data, and the second target coordinates.

[0011] In a possible implementation manner, the obtaining of the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system includes:

[0012] Obtain the first pose of the auxiliary target in the three-dimensional coordinate system of the shooting scene measured by a high-precision measuring instrument, and obtain the second pose of the main target in the three-dimensional coordinate system of the shooting scene measured by the high-precision measuring instrument.

[0013] Determine the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system according to the first pose and the second pose.

[0014] In a possible implementation manner, the performing of target key point detection on each piece of image data to respectively obtain the first image coordinates of the main target key points and the second image coordinates of the auxiliary target key points in each piece of image data includes:

[0015] Perform target detection on each piece of image data to respectively obtain each target in each piece of image data.

[0016] Perform target key point recognition on each target to respectively obtain the image coordinates of the target key points of each target.

[0017] Perform target recognition on the targets in each piece of image data, and respectively label the main target, the auxiliary target, the main target key points, and the auxiliary target key points in each piece of image data according to the target recognition result, wherein the image coordinates of the main target key points are the first image coordinates, and the image coordinates of the auxiliary target key points are the second image coordinates.

[0018] In a possible implementation, a unique target identification pattern is provided in each of the main target and the auxiliary target; identifying the targets in each of the image data, and respectively marking the main target, the auxiliary target, the key points of the main target, and the key points of the auxiliary target in each of the image data according to the target identification results, including:

[0019] For each image data, identify the unique target identification pattern of each target in the image data to obtain the target identification results of each target in the image data;

[0020] According to the target identification results of each target in the image data, mark the main target, the auxiliary target, the key points of the main target, and the key points of the auxiliary target in the image data.

[0021] In a possible implementation, identifying the targets in each of the image data, and respectively marking the main target, the auxiliary target, the key points of the main target, and the key points of the auxiliary target in each of the image data according to the target identification results, including:

[0022] Respectively determine the poses of each target in the lens coordinate system corresponding to each image data;

[0023] Select one image data from each of the image data to obtain the first image data, where the other image data except the first image data in each of the image data is the second image data;

[0024] Respectively set unique target identifiers for each target in the first image data to obtain the main target and the auxiliary target in the first image data;

[0025] For each second image data, traverse the matching relationship between the targets in the second image data and the targets in the first image data to obtain the relative poses of the camera under each matching relationship;

[0026] According to the poses of each target in the lens coordinate system corresponding to the second image data and the poses of each target in the lens coordinate system corresponding to the first image data, respectively calculate the errors of each relative pose;

[0027] According to the unique target identifiers of the main target and the auxiliary target in the first image data, mark the main target, the auxiliary target, the key points of the main target, and the key points of the auxiliary target in the second image data according to the matching relationship of the relative pose with the smallest error.

[0028] In a possible implementation, each of the image data is captured by the same lens of the camera at different positions;

[0029] Calibrating the parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates includes:

[0030] Calibrating the internal parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates.

[0031] In a possible implementation, the at least two image data include third image data collected by the first lens of the camera at a first position, and fourth image data collected by the second lens of the camera at a second position;

[0032] Calibrating the parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates includes:

[0033] Calibrating the transformation parameters of the lens coordinate system of the first lens and the lens coordinate system of the second lens according to the third conversion relationship corresponding to the third image data, the second image coordinates in the third image data, the third conversion relationship corresponding to the fourth image data, the second image coordinates in the fourth image data, and the second target coordinates.

[0034] In a second aspect, the present application provides a camera calibration device, including:

[0035] A first data acquisition module, configured to acquire at least two image data of a target array captured by a camera, and acquire first target coordinates of the main target key points of the main target in the main target coordinate system; wherein, the camera includes at least one lens, each of the image data is captured by the lens at different positions, the target array includes the main target and at least one auxiliary target, and the poses of the main target and the auxiliary target remain unchanged during the image data capture process;

[0036] A second data acquisition module, configured to, for each auxiliary target, acquire a first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system, and acquire second target coordinates of the auxiliary target key points of the auxiliary target in the auxiliary target coordinate system of the auxiliary target;

[0037] A key point detection module, configured to perform target key point detection on each of the image data, and respectively obtain first image coordinates of the main target key points and second image coordinates of the auxiliary target key points in each image data;

[0038] The first conversion relationship determination module is used to determine, for each piece of image data, a second conversion relationship between the main target coordinate system and the lens coordinate system corresponding to the image data according to the first image coordinates in the image data and the first target coordinates; wherein, the lens coordinate system corresponding to the image data is the lens coordinate system when the camera captures the image data.

[0039] The second conversion relationship determination module is used to determine, according to the second conversion relationship and the first conversion relationship, a third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data.

[0040] The parameter calibration module is used to calibrate the parameters of the camera according to the third conversion relationship corresponding to each piece of image data, the second image coordinates in each piece of image data, and the second target coordinates.

[0041] In a possible implementation manner, the second data acquisition module is specifically used for:

[0042] Obtain the first pose of the auxiliary target in the three-dimensional coordinate system of the shooting scene measured by a high-precision measuring instrument, and obtain the second pose of the main target in the three-dimensional coordinate system of the shooting scene measured by the high-precision measuring instrument;

[0043] Determine a first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system according to the first pose and the second pose.

[0044] In a possible implementation manner, the key point detection module includes:

[0045] The target detection sub-module is used to perform target detection on each piece of image data to obtain each target of each piece of image data respectively;

[0046] The key point recognition sub-module performs target key point recognition on each target to obtain the image coordinates of the target key points of each target respectively;

[0047] The target recognition sub-module is used to perform target recognition on the targets in each piece of image data, and respectively label the main target, the auxiliary target, the main target key points, and the auxiliary target key points in each piece of image data according to the target recognition results, wherein the image coordinates of the main target key points are the first image coordinates, and the image coordinates of the auxiliary target key points are the second image coordinates.

[0048] In a possible implementation manner, a unique target identification pattern is set in each of the main target and the auxiliary target; the target recognition sub-module is specifically used for:

[0049] For each image data, identify the unique target identification pattern for each target in the image data to obtain the target identification results of each target in the image data;

[0050] According to the target identification results of each target in the image data, mark the main target, auxiliary target, main target key points, and auxiliary target key points in the image data.

[0051] In a possible implementation manner, the target identification sub-module is specifically configured to:

[0052] Determine the poses of each target in the lens coordinate system corresponding to each image data respectively;

[0053] Select one image data from each of the image data to obtain the first image data, where the other image data except the first image data in each of the image data is the second image data;

[0054] Set unique target identifications for each target in the first image data respectively to obtain the main target and auxiliary target in the first image data;

[0055] For each second image data, traverse the matching relationship between the targets in the second image data and the targets in the first image data to obtain the relative poses of the camera under each matching relationship;

[0056] According to the poses of each target in the lens coordinate system corresponding to the second image data and the poses of each target in the lens coordinate system corresponding to the first image data, calculate the errors of each relative pose respectively;

[0057] According to the unique target identifications of the main target and auxiliary target in the first image data, mark the main target, auxiliary target, main target key points, and auxiliary target key points in the second image data according to the matching relationship of the relative pose with the smallest error.

[0058] In a possible implementation manner, each of the image data is captured by the same lens of the camera at different positions;

[0059] The parameter calibration module is specifically configured to:

[0060] Calibrate the internal parameters of the camera according to the third conversion relationship corresponding to each image data, the second image coordinates in each image data, and the second target coordinates.

[0061] In a possible implementation manner, the at least two image data include the third image data collected by the first lens of the camera at the first position and the fourth image data collected by the second lens of the camera at the second position;

[0062] The parameter calibration module is specifically configured to:

[0063] Calibrate the transformation parameters of the lens coordinate system of the first lens and the lens coordinate system of the second lens according to the third transformation relationship corresponding to the third image data, the second image coordinates in the third image data, the third transformation relationship corresponding to the fourth image data, the second image coordinates in the fourth image data, and the second target coordinates.

[0064] In a third aspect, the present application provides an electronic device, including:

[0065] A memory for storing a computer program;

[0066] A processor, when executing the program stored on the memory, implements any one of the camera calibration methods described in the present application.

[0067] In a fourth aspect, the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any one of the camera calibration methods described in the present application.

[0068] In a fifth aspect, an embodiment of the present application further provides a computer program product containing instructions, which when running on a computer, causes the computer to execute any one of the camera calibration methods described in the present application.

[0069] Beneficial effects of the embodiments of the present application:

[0070] The camera calibration method provided by the embodiments of the present application captures a target array composed of multiple two-dimensional targets with fixed relative positions, and uses the fixed relative position relationship between the main target and the auxiliary target. Only by determining the conversion relationship between the main target coordinate system and the auxiliary target coordinate system, and the conversion relationship between the main target coordinate system and the lens coordinate system, can the calibration of camera parameters be achieved. Compared with the calibration methods in the related art, the embodiments of the present application can be realized by selecting ordinary two-dimensional targets that are easily obtained to form a target array, reducing the resources required for camera calibration and making the calibration process simple and flexible to operate. By fixing the pose of the target, the calculation of the degrees of freedom of the target position is reduced, the error of the calibration result that may be caused is reduced, and the accuracy of the calibration result is improved. Since the pose of each target relative to the camera is independent and not related, and the positioning accuracy of the target has a significant impact on the target pose, the direct calculation of the conversion relationship between the auxiliary target coordinate system and the lens coordinate system is reduced during the calibration process. Only by constructing strong constraints on the relationship between the targets can the required calculation process be effectively reduced, the degrees of freedom of camera calibration be reduced, not only the robustness of the calibration result be constrained and the accuracy of the calibration result be improved, but also the efficiency of camera calibration can be further improved. In addition, the camera calibration process of the embodiments of the present application is not limited to the type of camera and can be applied to monocular cameras, binocular cameras, multi-camera systems, etc., improving the applicability of the camera calibration process.

[0071] Of course, it is not necessary for any product or method implementing the present application to achieve all the above advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0073] Figure 1-1 It is a schematic flowchart of a camera calibration method provided by the present application;

[0074] Figure 1-2 It is an example diagram of a target array provided by the present application;

[0075] Figure 1-3 It is an example diagram of image data captured by lenses at different positions provided by the present application;

[0076] Figure 1-4 It is an example diagram of images captured by lenses at different positions provided by the present application;

[0077] Figure 1-5 It is another example diagram of images captured by lenses at different positions provided by the present application;

[0078] Figure 1-6 A calculation example diagram of the camera calibration process in a related art provided for this application;

[0079] Figure 1-7 A calculation example diagram of the camera calibration process provided for this application;

[0080] Figure 2 A possible implementation manner of step S12 provided for this application;

[0081] Figure 3 A possible implementation manner of step S13 provided for this application;

[0082] Figure 4-1 A possible implementation manner of step S33 provided for this application;

[0083] Figure 4-2 An example diagram of image data of each target shot by lenses at different positions provided for this application;

[0084] Figure 4-3 Another example diagram of image data of each target shot by lenses at different positions provided for this application;

[0085] Figure 5 Another possible implementation manner of step S33 provided for this application;

[0086] Figure 6 A schematic structural diagram of a camera calibration device provided for this application;

[0087] Figure 7 A schematic structural diagram of an electronic device provided for this application. Specific embodiments

[0088] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art based on this application belong to the scope of protection of this application.

[0089] Since, in the traditional camera calibration process, in order to meet the camera calibration accuracy within the field of view, it is usually necessary to use a relatively large target board, calibration tooling, etc. as three-dimensional space objects for calibration. Such target boards, calibration tooling, etc. are often large in size and difficult to ensure the machining accuracy, and are not easily obtained. However, if an ordinary two-dimensional target (i.e., a planar target) is used, it may lead to an overly complex calibration process and difficult to ensure the calibration accuracy, thereby resulting in a low calibration efficiency of the camera. To solve at least one of the above problems, the present application provides a camera calibration method, device, and electronic device, which will be described in detail below through specific embodiments.

[0090] First, the specific terms in the embodiments of the present application are explained:

[0091] Two-dimensional target: A planar target, including a virtual coordinate system of the target and feature points thereon. The feature points can be circular points, corner points, or other feature points that can be detected by a fixed image detection algorithm.

[0092] Nonlinear optimization: A mathematical method that abstracts a problem into a multivariate nonlinear function and obtains the solution when the minimum / maximum value of the nonlinear function is obtained through numerical or analytical methods as the optimal solution of the problem.

[0093] In the first aspect, referring to Figure 1-1 , a flowchart of a camera calibration method provided by the present application includes:

[0094] Step S11: Obtain at least two image data of a target array captured by a camera, and obtain the first target coordinates of the main target key points of the main target in the main target coordinate system;

[0095] Among them, the camera includes at least one lens, that is, the camera is a monocular camera including one lens or a multiocular camera including multiple lenses (such as a binocular camera, a trinocular camera), and can also be a structured light depth sensor system. Each image data is captured by the lens at different positions. Specifically, when the camera is a monocular camera, each image data is respectively captured when the camera is placed at different positions; when the camera is a multiocular camera, each image data can be respectively captured by multiple lenses of the multiocular camera, or can also be captured by different lenses when the multiocular camera is placed at different positions. The target array includes a main target and at least one auxiliary target, that is, it includes one main target and one or more auxiliary targets, and the poses of the main target and the auxiliary target remain unchanged during the image data capture process.

[0096] It should be emphasized that there is no substantial primary-secondary relationship between the primary target and the secondary targets. The primary target and the secondary targets are only used as a logical distinction. Specifically, the primary target can be randomly selected from the target array. For example, the primary target can be the target at the leftmost position, or the target at the center position, or the target at the rightmost position, etc. After the primary target is selected, the other targets in the target array are called secondary targets.

[0097] Both the primary target and the secondary targets are two-dimensional planar targets. Each target has its own target coordinate system, and the target includes feature points as key target points. The primary target has a primary target coordinate system and primary target key points, and the secondary targets have secondary target coordinate systems and secondary target key points. Specifically, the key target points can be dot points, corner points, or other feature points that can be detected by an image detection algorithm, and can be specifically selected according to actual needs. Since the target has an area, in the actual processing, the key target points can be used to represent the corresponding target. For the selection of the key target points, it can be achieved by any feature point extraction algorithm, such as a dot point extraction algorithm, a checkerboard target corner point extraction algorithm, etc.

[0098] In one example, the primary target and the secondary targets can both be checkerboard targets with triangular markings. For example Figure 1-2 the shown target array, which includes a checkerboard primary target at the center position and four checkerboard secondary targets at the surrounding positions. The triangular markings on the targets represent the origin of the target coordinate system, and the orientation of the triangular markings represents the direction of the target coordinate system. Specifically, through any of the above feature extraction algorithms, obtain the feature point coordinates of the primary target, then identify the triangular marking on the primary target as the origin, and the orientation of the triangular marking as the orientation of the target coordinate system. Then, determine the physical coordinates of the primary target key points in the primary target coordinate system through corner point sorting to obtain the first target coordinates.

[0099] Shoot two or more image data of the target array through lenses at different positions. For example Figure 1-3 as shown, shoot the target array through the lenses located at POS1 (position 1), POS2 (position 2), and POS3 (position 3) respectively. The image data obtained by shooting at POS1 is as Figure 1-4 shown, and the image data obtained by shooting at POS3 is as Figure 1-5 shown.

[0100] Step S12: For each secondary target, obtain the first conversion relationship between the secondary target coordinate system of the secondary target and the primary target coordinate system, and obtain the second target coordinates of the secondary target key points of the secondary target in the secondary target coordinate system of the secondary target.

[0101] The same processing procedure is performed for each secondary target. Among them, the coordinates of the key points of each secondary target in its respective secondary target coordinate system are obtained to get their respective second target coordinates. In one example, the process of obtaining the second target coordinates can be the same as the process of obtaining the first target coordinates of the key points of the main target in the above text.

[0102] In one example, the first conversion relationship between the secondary target coordinate system and the main target coordinate system can first obtain the rotation matrix and translation vector between the secondary target coordinate system and the main target coordinate system based on the respective origins and coordinate axes of the secondary target coordinate system and the main target coordinate system, and then establish it based on the obtained rotation matrix and translation vector. The determination of the first conversion relationship can refer to any coordinate system conversion algorithm in the related art and is not limited here.

[0103] The first conversion relationship can be used to represent the relative relationship of the secondary target coordinate system with respect to the main target coordinate system, and can also be used to represent the relative relationship of the secondary target with respect to the main target coordinate system. For the key points of the secondary target and the key points of the main target, through the first conversion relationship and their respective coordinates in their own target coordinate systems, the conversion coordinates in the target coordinate system of the other party can be obtained. For example, the key points of the secondary target can obtain the conversion coordinates of the secondary target in the main target coordinate system according to the second target coordinates and the first conversion relationship.

[0104] Step S13: Detect the key points of the target for each piece of the image data, and respectively obtain the first image coordinates of the key points of the main target and the second image coordinates of the key points of the secondary target in each piece of image data.

[0105] The key points of the main target and the key points of the secondary target exist in each piece of image data, and each piece of image data has its own lens coordinate system. In the embodiments of the present application, the key points of the target are detected for each piece of image data to obtain the first image coordinates of the main target in the lens coordinate system of each piece of image data, and the second image coordinates of the secondary target in the lens coordinate system of each piece of image data. The detection of the key points of the target in the image data can also be implemented according to any feature point extraction algorithm.

[0106] Step S14: For each piece of image data, determine the second conversion relationship between the main target coordinate system and the lens coordinate system corresponding to this piece of image data according to the first image coordinates in this piece of image data and the first target coordinates;

[0107] Among them, the lens coordinate system corresponding to this piece of image data is the lens coordinate system when the camera captures this piece of image data.

[0108] The lens coordinate system corresponding to each image data is the lens coordinate system when the camera takes pictures. As mentioned above, each image data is obtained by taking pictures with lenses at different positions. Therefore, the lens coordinate system of each image data is different and can also represent the camera coordinate system at the corresponding position.

[0109] According to the first image coordinates of the main target key points in the corresponding lens coordinate system in each image data and the first target coordinates of the main target key points in the main target coordinate system, the second conversion relationship between the main target coordinate system and the lens coordinate system is determined. The determination of the second conversion relationship can be realized by referring to any image coordinate system - camera coordinate system conversion algorithm. For example, based on the first image coordinates and the first target coordinates, the rotation matrix and translation vector between the main target coordinate system and the lens coordinate system are determined, so as to obtain the second conversion relationship between the main target coordinate system and the lens coordinate system.

[0110] The second conversion relationship can be used to represent the relative relationship of the main target with respect to the lens position corresponding to each image data.

[0111] Step S15: Determine the third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data according to the second conversion relationship and the first conversion relationship.

[0112] In the embodiments of the present application, based on the first conversion relationship between the auxiliary target coordinate system and the main target coordinate system and the second conversion relationship between the main target coordinate system and the lens coordinate system, the third conversion relationship between the auxiliary target coordinate system and the lens coordinate system can be directly obtained, without calculating the conversion relationship between the auxiliary target coordinate system and the lens coordinate system through the second target coordinates of the auxiliary target in the auxiliary target coordinate system and its second image coordinates in the lens coordinate system.

[0113] The third conversion relationship can be used to represent the relative relationship of the auxiliary target with respect to the lens position corresponding to each image data, and also to represent the relative relationship of the auxiliary target coordinate system with respect to the lens coordinate system.

[0114] Step S16: Calibrate the parameters of the camera according to the third conversion relationship corresponding to each image data, the second image coordinates and the second target coordinates in each image data.

[0115] In the embodiments of the present application, the Zhang Zhengyou camera parameter calibration method in the related technology is introduced first: First, it is defaulted that the camera satisfies the projection model and distortion model of the camera, that is, the internal parameter matrix A and distortion parameter k of the camera are expressed as:

[0116]

[0117] where, f x 、f yIndicates the camera focal lengths in the x and y directions in the lens coordinate system, with the unit being pixels; c x , c y Indicates the coordinates of the camera principal point (the intersection of the camera optical axis and the imaging plane) in the target coordinate system, with the unit being pixels; k1, k2, and k3 are the camera radial distortion coefficients, and p1, p2 are the tangential distortion coefficients.

[0118] During the Zhang Zhengyou calibration process, each captured target is regarded as an individual target (without distinguishing between the main target and the auxiliary target). Based on the physical coordinates of each target key point in the target coordinate system and its image coordinates in the lens coordinate system, an initial estimate of the camera internal parameter A is obtained, as well as the pose information of each target relative to the shooting lens position corresponding to each image data, such as the rotation matrix R and the translation vector t. For any target key point P in each image data t If its target coordinates in its own target coordinate system are p t , then its coordinates p c in the lens coordinate system can be expressed as:

[0119]

[0120] where R t , t t represent the rotation matrix and the translation vector of the target where P is located relative to the lens coordinate system.

[0121] Combined with the model, the coordinates t onto which the target key point P is projected onto the camera image plane can be expressed as:

[0122]

[0123] where (x d , y d ) are the normalized distorted coordinates, expressed as:

[0124]

[0125] where (x n , y n ) are the normalized coordinates, expressed as:

[0126]

[0127] Meanwhile, the image coordinates of the above target key point in the lens coordinate system are (u i , v i ), and the optimization objective function for camera parameter calibration is the sum of the reprojection errors of all target key points of all targets, which can be expressed as:

[0128]

[0129] Then, an optimization algorithm is adopted to implement the non - linear optimization of the camera parameter calibration, and the distortion parameter k and the camera internal parameter A that minimize F are obtained, completing the parameter optimization of the camera calibration parameters and realizing the camera calibration process. c It can be understood that in the camera calibration process in the related art, it is necessary to directly calculate the relative relationship of each target to the lens coordinate system through the physical coordinates of each target in the target coordinate system and the image coordinates of each target in the lens coordinate system, as shown in, for example

[0130] shown. In the embodiment of the present application, taking the main target as the reference standard, the relative matrix (the third conversion relationship) of the auxiliary target relative to the lens coordinate system can be expressed as: Figure 1-6 shown. In the embodiment of the present application, taking the main target as the reference standard, the relative matrix (the third conversion relationship) of the auxiliary target relative to the lens coordinate system can be expressed as:

[0131]

[0132] where S qo is the relative matrix (the first conversion relationship) of the auxiliary target q relative to the main target coordinate system of the main target o, is the relative matrix (the second conversion relationship) of the main target o relative to the lens coordinate system. Then, it can be calculated only based on the relative relationship of the main target to the lens position corresponding to each image data and the relative relationship of the auxiliary target coordinate system to the main target coordinate system, as shown in, for example Figure 1-7 shown. Specifically, the above - mentioned first conversion relationship and second conversion relationship can be used to replace the rotation matrix and translation vector in formula (2) above. Based on this, the parameter optimization of the camera calibration parameters can be completed, and the camera calibration process can be realized.

[0133] In one embodiment of the present application, each of the image data is captured by the same lens of the camera at different positions;

[0134] The above - mentioned step S16 calibrates the parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates, including:

[0135] Calibrating the internal parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates.

[0136] In the embodiment of the present application, the camera is a monocular camera. Then, each image data is captured by the same lens of the camera at different positions. Therefore, the parameter calibration of the camera is the calibration of the camera internal parameters, further improving the efficiency of monocular camera calibration.

[0137] In one embodiment of the present application, the at least two image data include third image data collected by a first lens of the camera at a first position, and fourth image data collected by a second lens of the camera at a second position;

[0138] The above step S16 calibrates the parameters of the camera according to the third conversion relationships corresponding to the respective image data, the second image coordinates in the respective image data, and the second target coordinates, and includes:

[0139] Calibrate the transformation parameters of the lens coordinate system of the first lens and the lens coordinate system of the second lens according to the third conversion relationship corresponding to the third image data, the second image coordinates in the third image data, the third conversion relationship corresponding to the fourth image data, the second image coordinates in the fourth image data, and the second target coordinates.

[0140] In an embodiment of the present application, the camera is a multi-camera, that is, a camera including two or more lenses. For a multi-camera, it is necessary to calibrate between two lenses pairwise. Two of the lenses are respectively used as the first lens and the second lens, and calibration is performed between the first lens and the second lens. The image data includes third image data collected by the first lens at a first position and fourth image data collected by the second lens at a second position. The first position and the second position may be two positions where the first lens and the second lens are respectively located when the camera is fixed, or may be two positions where the first lens and the second lens are respectively located when the camera is at different positions during the movement and transformation of the camera at different positions. At this time, the calibration of the camera parameters is the calibration of the transformation parameters of the lens coordinate system of the first lens and the lens coordinate system of the second lens. Further improves the accuracy and robustness of multi-camera calibration.

[0141] As can be seen from the above, in the camera calibration method provided by the embodiment of the present application, first, the targets in the target array are divided into main targets and auxiliary targets, and then at least two image data of the target array captured by the camera are obtained, as well as the first target coordinates of the key points of the main target in the main target coordinate system; then for each auxiliary target, the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system is obtained, and the second target coordinates of the key points of the auxiliary target in the auxiliary target coordinate system of the auxiliary target are obtained. Target key point detection is performed on each image data to respectively obtain the first image coordinates of the key points of the main target and the second image coordinates of the key points of the auxiliary target in each image data; then for each image data, according to the first image coordinates and the first target coordinates in the image data, the second conversion relationship between the main target coordinate system and the lens coordinate system corresponding to the image data is determined; and then according to the second conversion relationship and the first conversion relationship, the third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data is determined. Finally, according to the third conversion relationship corresponding to each image data, the second image coordinates and the second target coordinates in each image data, the parameters of the camera are calibrated.

[0142] In the embodiment of the present application, by photographing a target array composed of a plurality of two-dimensional targets with fixed relative positions, and using the fixed relative position relationship between the main target and the auxiliary target, it is only necessary to determine the conversion relationship between the main target coordinate system and the auxiliary target coordinate system, and the conversion relationship between the main target coordinate system and the lens coordinate system, so as to realize the calibration of the camera parameters. Compared with the calibration method in the related art, the embodiment of the present application can be realized by selecting an easily obtained ordinary two-dimensional target to form a target array, reducing the resources required for camera calibration, and making the calibration process simple and flexible. By fixing the pose of the target, the calculation of the degrees of freedom of the target position is reduced, the error of the calibration result that may be caused is reduced, and the accuracy of the calibration result is improved. Since the pose of each target relative to the camera is independent and not related, and the positioning accuracy of the target has a significant impact on the target pose, the direct calculation of the conversion relationship between the auxiliary target coordinate system and the lens coordinate system is reduced during the calibration process, and only by constructing a strong constraint on the relationship between the targets can the required calculation process be effectively reduced, reducing the degrees of freedom of camera calibration, not only constraining the robustness of the calibration result, improving the accuracy of the calibration result, but also further improving the efficiency of camera calibration. In addition, the camera calibration process of the embodiment of the present application is not limited to the type of camera, and can be applied to monocular cameras, binocular cameras, multi-camera cameras, etc., improving the applicability of the camera calibration process.

[0143] In a possible implementation manner, as Figure 2 shown, the above step S12 of obtaining the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system includes:

[0144] Step S21: Obtain the first pose of the auxiliary target in the three-dimensional coordinate system of the shooting scene measured by a high-precision measuring instrument, and obtain the second pose of the main target in the three-dimensional coordinate system of the shooting scene measured by the high-precision measuring instrument;

[0145] Step S22: Determine the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system according to the first pose and the second pose.

[0146] In the embodiment of the present application, the poses of the auxiliary target and the main target in the three-dimensional coordinate system (i.e., the world coordinate system) of the shooting scene are respectively measured by a high-precision measuring instrument to obtain the first pose and the second pose. According to the first pose and the second pose, the first conversion relationship between the auxiliary target coordinate system and the main target coordinate system can be directly obtained. In one example, the high-precision measuring instrument may be a three-dimensional measuring coordinate instrument. In another example, the relative poses between the main target and the auxiliary target can also be directly fixed by a high-precision fixed structure. At this time, the processing parameters of the fixed structure can be directly used as the relative relationship between the main target and the auxiliary target to obtain the first conversion relationship.

[0147] As can be seen from the above, in the camera calibration method provided by the embodiment of the present application, the poses of the auxiliary target and the main target in the three-dimensional coordinate system of the shooting scene are directly obtained by a high-precision measuring instrument. Based on this, the first conversion relationship between the auxiliary target coordinate system and the main target coordinate system can be obtained, and the high precision of the high-precision measuring instrument is used to further improve the accuracy of camera calibration.

[0148] In a possible implementation manner, as Figure 3 shown, the above step S13 performs target key point detection on each of the image data, and respectively obtains the first image coordinates of the main target key points and the second image coordinates of the auxiliary target key points in each image data, including:

[0149] Step S31: Perform target detection on each of the image data to respectively obtain each target in each image data;

[0150] Step S32: Perform target key point recognition on each of the targets to respectively obtain the image coordinates of the target key points of each of the targets;

[0151] Step S33: Perform target recognition on the targets in each of the image data, and respectively label the main target, the auxiliary target, the main target key points, and the auxiliary target key points in each of the image data according to the target recognition results;

[0152] Among them, the image coordinates of the main target key points are the first image coordinates, and the image coordinates of the auxiliary target key points are the second image coordinates.

[0153] It can be understood that after images are captured for the target array using lenses at different positions, the angles of the target array in the obtained image data can be different, and the positions of the target array in the image data can also be different. Therefore, it is necessary to determine consistent main targets and auxiliary targets in each of the obtained image data to avoid confusion that may be caused by different positions of the targets in different image data.

[0154] In the embodiments of the present application, first, target detection is performed on each image data to obtain each target in each image data respectively. Then, target key point recognition is performed on each target to obtain the image coordinates of the target key points of each target respectively, which can be implemented based on any image feature point recognition algorithm here. Then, target recognition is performed on the targets in each image data, and the main target, auxiliary target, main target key points, and auxiliary target key points in each image data are respectively marked according to the target recognition results, so that the main target, auxiliary target, main target key points, and auxiliary target key points are consistent in different image data.

[0155] As can be seen from the above, in the camera calibration method provided by the embodiments of the present application, by performing target recognition on the targets in each image data, the main target, auxiliary target, main target key points, and auxiliary target key points are consistent in different image data, avoiding confusion that may be caused by different positions of the targets in different image data, and further improving the accuracy of camera calibration.

[0156] In a possible implementation manner, as Figure 4-1 shown, a unique target identification pattern is respectively set in the above-mentioned main target and the auxiliary target; the above-mentioned step S33 performs target recognition on the targets in each of the image data, and respectively marks the main target, auxiliary target, main target key points, and auxiliary target key points in each of the image data according to the target recognition results, including:

[0157] Step S41: For each image data, perform recognition on the unique target identification pattern of each target in this image data to obtain the target recognition results of each target in this image data;

[0158] Step S42: According to the target recognition results of each target in this image data, mark the main target, auxiliary target, main target key points, and auxiliary target key points in this image data.

[0159] In the embodiments of the present application, before image data is captured for the target array, a unique target identification pattern is respectively set in the main target and the auxiliary target. For example, different serial numbers are marked on each target, different special symbols are marked on each target, etc. As Figure 4-2 and Figure 4-3As shown, it is an example of each piece of image data captured when each target has a unique target identification pattern. For each piece of image data, the unique target identification pattern of each target is identified to obtain the target identification result of each target in the image data. Based on this, the main target, auxiliary target, main target key points, and auxiliary target key points in each piece of image data are marked.

[0160] As can be seen from the above, the camera calibration method provided by the embodiments of the present application, by pre-setting a unique target identification pattern for each target before shooting, and then identifying the unique target identification pattern in the captured image data, obtains a target identification result that can directly mark the main target, auxiliary target, main target key points, and auxiliary target key points, further improving the accuracy and efficiency of camera calibration.

[0161] In a possible implementation manner, as Figure 5 shown, the above step S33 performs target identification on the targets in each of the image data, and respectively marks the main target, auxiliary target, main target key points, and auxiliary target key points in each of the image data according to the target identification result, including:

[0162] Step S51: Determine the poses of each of the targets in the lens coordinate system corresponding to each of the image data respectively;

[0163] Step S52: Select one piece of image data from each of the image data to obtain the first image data;

[0164] Among them, the other image data except the first image data in each of the image data is the second image data;

[0165] Step S53: Set a unique target identification for each of the targets in the first image data respectively to obtain the main target and auxiliary target in the first image data;

[0166] Step S54: For each second image data, traverse the matching relationship between the targets in the second image data and the targets in the first image data to obtain the relative pose of the camera under each matching relationship;

[0167] Step S55: Calculate the errors of each of the relative poses respectively according to the poses of each of the targets in the lens coordinate system corresponding to the second image data and the poses of each of the targets in the lens coordinate system corresponding to the first image data;

[0168] Step S56: According to the unique target identifications of the main target and auxiliary target in the first image data, mark the main target, auxiliary target, main target key points, and auxiliary target key points in the second image data according to the matching relationship of the relative pose with the smallest error.

[0169] In the embodiments of the present application, first, the poses of each target in the lens coordinate system corresponding to each image data are determined respectively. In one example, it can be determined based on the method in Zhang Zhengyou calibration method. Specifically, given the coordinates of each target in its respective target coordinate system and its respective image coordinates, the homography matrix between each target and the lens corresponding to each image data can be determined. Then, by performing singular value decomposition and matrix recombination on the homography matrix, the internal parameter matrix of the camera can be calculated. Furthermore, through the internal parameter matrix and the homography matrix of each target with respect to the lens, the poses of each target in the lens coordinate system corresponding to each image data can be obtained.

[0170] Then, a unique target identifier is set for each target in the first image data, and each target in the second image data is identified and determined with reference to the first image data. Specifically, the first image data can be the image data captured by the lens at the most frontal position of the target array, or the first captured image data.

[0171] Then, the matching relationship between the targets in the second image data and the targets in the first image data is traversed, that is, a matching relationship is established for any pair of targets in the second image data and the first image data. The matching relationship indicates that the pair of targets with the matching relationship is the same target in different image data. At this time, multiple matching relationships can be obtained. For each matching relationship, based on the poses of the targets with the matching relationship in the lens coordinate system corresponding to each image data, the relative pose of the camera under this matching relationship is calculated. The relative pose of the camera represents the relative relationship between the lens coordinate systems corresponding to different image data.

[0172] Then, each target in the first image data is transformed to the lens coordinate system corresponding to the second image data based on the relative pose to obtain the transformed pose of each target. For the transformed pose of each target in the first image data and the pose of the target in the second image data with a matching relationship with it in the lens coordinate system corresponding to the second image data, the error is calculated. Then, the errors of each target are summed to obtain the error of the relative pose. The relative pose with the minimum error is considered the optimal pose. Then, according to the unique target identifiers of the main target and the auxiliary target in the first image data, and in accordance with the matching relationship of the relative pose (optimal pose) with the minimum error, the main target, the auxiliary target, the key points of the main target, and the key points of the auxiliary target in the second image data are marked, that is, it is considered that the matching relationship of the optimal pose is correct, and the targets with this matching relationship are the same target in different image data.

[0173] In one example, further, based on the optimal pose, each target in the first image data can be converted into the lens coordinate system corresponding to the second image data again, and the converted pose of each target can be obtained again. For each target in the first image data after conversion, the target in the second image data with the pose closest to its converted pose in the lens coordinate system corresponding to the second image data is considered to have a matching relationship, and a more accurate matching relationship can be obtained. Based on this, the main target, auxiliary target, main target key points, and auxiliary target key points are marked.

[0174] As can be seen from the above, in the camera calibration method provided by the embodiments of the present application, in the case where each target does not have a unique target identification pattern in advance, the relative pose between the lens coordinate systems corresponding to each image data can be determined to obtain the matching relationship of each target in each image data, so as to mark the main target, auxiliary target, main target key points, and auxiliary target key points in each image data, further improving the universality of the camera calibration method.

[0175] In a second aspect, referring to Figure 6 , a structural schematic diagram of a camera calibration device is further provided in the embodiments of the present application, including:

[0176] A first data acquisition module 601, configured to acquire at least two image data of a target array captured by a camera, and acquire the first target coordinates of the main target key points of the main target in the main target coordinate system; wherein, the camera includes at least one lens, each of the image data is captured by the lens at different positions, the target array includes the main target and at least one auxiliary target, and the poses of the main target and the auxiliary target remain unchanged during the image data capture process;

[0177] A second data acquisition module 602, configured to, for each auxiliary target, acquire the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system, and acquire the second target coordinates of the auxiliary target key points of the auxiliary target in the auxiliary target coordinate system of the auxiliary target;

[0178] A key point detection module 603, configured to perform target key point detection on each of the image data, and respectively obtain the first image coordinates of the main target key points and the second image coordinates of the auxiliary target key points in each image data;

[0179] A first conversion relationship determination module 604, configured to, for each image data, determine the second conversion relationship between the main target coordinate system and the lens coordinate system corresponding to the image data according to the first image coordinates in the image data and the first target coordinates; wherein, the lens coordinate system corresponding to the image data is the lens coordinate system when the camera captures the image data;

[0180] The second conversion relationship determination module 605 is configured to determine a third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data according to the second conversion relationship and the first conversion relationship;

[0181] The parameter calibration module 606 is configured to calibrate the parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates.

[0182] As can be seen from the above, the camera calibration device provided by the embodiment of the present application first divides the targets in the target array into a main target and auxiliary targets, and then acquires at least two pieces of image data of the target array captured by the camera, and the first target coordinates of the key points of the main target in the main target coordinate system; then for each auxiliary target, acquire the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system, and acquire the second target coordinates of the key points of the auxiliary target of the auxiliary target in the auxiliary target coordinate system of the auxiliary target. Detect the target key points in each piece of image data to respectively obtain the first image coordinates of the key points of the main target and the second image coordinates of the key points of the auxiliary target in each piece of image data; then for each piece of image data, determine the second conversion relationship between the main target coordinate system and the lens coordinate system corresponding to the image data according to the first image coordinates and the first target coordinates in the image data; and then determine the third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data according to the second conversion relationship and the first conversion relationship. Finally, calibrate the parameters of the camera according to the third conversion relationship corresponding to each piece of image data, the second image coordinates in each piece of image data, and the second target coordinates.

[0183] In the embodiment of the present application, by photographing a target array composed of multiple two-dimensional targets with fixed relative positions, and using the fixed relative position relationship between the main target and the auxiliary target, it is only necessary to determine the conversion relationship between the main target coordinate system and the auxiliary target coordinate system, and the conversion relationship between the main target coordinate system and the lens coordinate system, then the calibration of the camera parameters can be achieved. Compared with the calibration method in the related art, in the embodiment of the present application, an easily obtainable ordinary two-dimensional target is selected to form a target array, which reduces the resources required for camera calibration and makes the calibration process simple and flexible to operate. By fixing the pose of the target, the calculation of the degrees of freedom of the target position is reduced, the error of the calibration result that may be caused is reduced, and the accuracy of the calibration result is improved. Since the pose of each target relative to the camera is independent and uncorrelated, and the positioning accuracy of the target has a significant impact on the target pose, the direct calculation of the conversion relationship between the auxiliary target coordinate system and the lens coordinate system is reduced during the calibration process. Only by constructing a strong constraint on the relationship between the targets can the required calculation process be effectively reduced, the degrees of freedom of camera calibration be reduced, not only the robustness of the calibration result be constrained and the accuracy of the calibration result be improved, but also the efficiency of camera calibration can be further improved. In addition, the camera calibration process of the embodiment of the present application is not limited to the type of camera, and can be applied to monocular cameras, binocular cameras, multi-camera cameras, etc., improving the applicability of the camera calibration process.

[0184] In one embodiment of the present application, the second data acquisition module 602 is specifically configured to:

[0185] Obtain the first pose of the auxiliary target in the three-dimensional coordinate system of the shooting scene measured by a high-precision measuring instrument, and obtain the second pose of the main target in the three-dimensional coordinate system of the shooting scene measured by the high-precision measuring instrument;

[0186] According to the first pose and the second pose, determine the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system.

[0187] As can be seen from the above, the camera calibration device provided by the embodiment of the present application directly obtains the poses of the auxiliary target and the main target in the three-dimensional coordinate system of the shooting scene through a high-precision measuring instrument, and based on this, the first conversion relationship between the auxiliary target coordinate system and the main target coordinate system can be obtained. Utilizing the high precision of the high-precision measuring instrument further improves the accuracy of camera calibration.

[0188] In one embodiment of the present application, the key point detection module 603 includes:

[0189] A target detection sub-module, configured to perform target detection on each of the image data, and respectively obtain each target of each image data;

[0190] The key point recognition sub-module performs target key point recognition on each of the targets, and respectively obtains the image coordinates of the target key points of each of the targets;

[0191] The target recognition sub-module is used to perform target recognition on the targets in each of the image data, and respectively mark the main target, auxiliary target, main target key points and auxiliary target key points in each of the image data according to the target recognition results. Among them, the image coordinates of the main target key points are the first image coordinates, and the image coordinates of the auxiliary target key points are the second image coordinates.

[0192] As can be seen from the above, the camera calibration device provided by the embodiment of the present application performs target recognition on the targets in each image data, so that the main target, auxiliary target, main target key points and auxiliary target key points are consistent in different image data, avoiding the confusion that may be caused by different positions of the targets in different image data, and further improving the accuracy of camera calibration.

[0193] In one embodiment of the present application, a unique target identification pattern is respectively set in each of the main target and the auxiliary target; the target recognition sub-module is specifically used for:

[0194] For each image data, perform recognition of the unique target identification pattern on each of the targets in this image data to obtain the target recognition results of each of the targets in this image data;

[0195] According to the target recognition results of each of the targets in this image data, mark the main target, auxiliary target, main target key points and auxiliary target key points in this image data.

[0196] As can be seen from the above, the camera calibration device provided by the embodiment of the present application pre-sets a unique target identification pattern for each target before shooting, and then recognizes the unique target identification pattern in the captured image data to obtain the target recognition results that can directly mark the main target, auxiliary target, main target key points and auxiliary target key points, further improving the accuracy and efficiency of camera calibration.

[0197] In one embodiment of the present application, the target recognition sub-module is specifically used for:

[0198] Respectively determine the poses of each of the targets in the lens coordinate system corresponding to each of the image data;

[0199] Select one image data from each of the image data to obtain the first image data, where the other image data except the first image data in each of the image data is the second image data;

[0200] Respectively set unique target identifiers for each of the targets in the first image data to obtain the main target and the auxiliary target in the first image data;

[0201] For each second image data, traverse the matching relationship between the targets in the second image data and the targets in the first image data to obtain the relative poses of the camera under each matching relationship;

[0202] According to the poses of the targets in the lens coordinate system corresponding to the second image data and the poses of the targets in the lens coordinate system corresponding to the first image data, calculate the errors of the relative poses respectively;

[0203] According to the unique target identifiers of the main target and the auxiliary targets in the first image data, label the main target, the auxiliary target, the key points of the main target and the key points of the auxiliary target in the second image data according to the matching relationship of the relative pose with the smallest error.

[0204] As can be seen from the above, the camera calibration device provided by the embodiment of the present application can, in the case where the targets do not have unique target identification patterns in advance, also obtain the matching relationships of the targets in each image data by determining the relative poses between the lens coordinate systems corresponding to the image data, so as to label the main target, the auxiliary target, the key points of the main target and the key points of the auxiliary target in each image data, further improving the universality of the camera calibration method.

[0205] In an embodiment of the present application, each of the image data is captured by the same lens of the camera at different positions;

[0206] The parameter calibration module 606 is specifically configured to:

[0207] Calibrate the internal parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates.

[0208] The camera calibration device provided by the embodiment of the present application further improves the efficiency of monocular camera calibration.

[0209] In an embodiment of the present application, the at least two image data include third image data collected by the first lens of the camera at a first position and fourth image data collected by the second lens of the camera at a second position;

[0210] The parameter calibration module 606 is specifically configured to:

[0211] Calibrate the transformation parameters of the lens coordinate system of the first lens and the lens coordinate system of the second lens according to the third conversion relationship corresponding to the third image data, the second image coordinates in the third image data, the third conversion relationship corresponding to the fourth image data, the second image coordinates in the fourth image data, and the second target coordinates.

[0212] The camera calibration device provided by the embodiments of the present application further improves the accuracy and robustness of multi-camera calibration.

[0213] The embodiments of the present application also provide an electronic device, as Figure 7 shown, including:

[0214] A memory 701 for storing a computer program;

[0215] A processor 702, when executing the program stored on the memory 701, implements the steps of the camera calibration method described in any one of the above.

[0216] And the above-mentioned electronic device may further include a communication bus and / or a communication interface. The processor 702, the communication interface, and the memory 701 complete communication with each other through the communication bus.

[0217] The communication bus mentioned in the above-mentioned electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0218] The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0219] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0220] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0221] In another embodiment provided by the present application, a computer-readable storage medium is further provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above camera calibration methods are implemented.

[0222] In another embodiment provided by the present application, a computer program product including instructions is further provided. When it runs on a computer, the computer is caused to execute any of the camera calibration methods in the above embodiments.

[0223] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), etc.

[0224] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0225] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0226] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A camera calibration method, characterized in that, Including: Obtaining at least two image data of a target array captured by a camera, and obtaining first target coordinates of key points of a main target in a main target coordinate system of the main target; wherein, the camera includes at least one lens, each of the image data is captured by the lens at different positions, the target array includes the main target and at least one auxiliary target, and the poses of the main target and the auxiliary target remain unchanged during the process of capturing the image data; For each auxiliary target, obtaining a first conversion relationship between an auxiliary target coordinate system of the auxiliary target and the main target coordinate system, and obtaining second target coordinates of key points of the auxiliary target in the auxiliary target coordinate system of the auxiliary target; Performing target key point detection on each of the image data to respectively obtain first image coordinates of key points of the main target and second image coordinates of key points of the auxiliary target in each image data; For each image data, determining a second conversion relationship between the main target coordinate system and a lens coordinate system corresponding to the image data according to the first image coordinates in the image data and the first target coordinates; wherein, the lens coordinate system corresponding to the image data is the lens coordinate system when the camera captures the image data; Determining a third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data according to the second conversion relationship and the first conversion relationship; Calibrating parameters of the camera according to the third conversion relationships corresponding to the respective image data, the second image coordinates in the respective image data, and the second target coordinates.

2. The method according to claim 1, wherein The obtaining of the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system includes: Obtaining a first pose of the auxiliary target in a three-dimensional coordinate system of a shooting scene measured by a high-precision measuring instrument, and obtaining a second pose of the main target in the three-dimensional coordinate system of the shooting scene measured by the high-precision measuring instrument; Determining the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system according to the first pose and the second pose.

3. The method according to claim 1, wherein The performing of target key point detection on each of the image data to respectively obtain first image coordinates of key points of the main target and second image coordinates of key points of the auxiliary target in each image data includes: Performing target detection on each of the image data to respectively obtain each target in each image data; Performing target key point recognition on each of the targets to respectively obtain image coordinates of key points of each of the targets; Performing target recognition on the targets in each of the image data, and respectively marking the main target, the auxiliary target, the key points of the main target, and the key points of the auxiliary target in each of the image data according to the target recognition results, wherein, the image coordinates of the key points of the main target are the first image coordinates, and the image coordinates of the key points of the auxiliary target are the second image coordinates.

4. The method according to claim 3, characterized in that, A unique target identification pattern is respectively set in the main target and the auxiliary target; the performing of target recognition on the targets in each of the image data, and respectively marking the main target, the auxiliary target, the key points of the main target, and the key points of the auxiliary target in each of the image data according to the target recognition results includes: For each image data, identify the unique target identification pattern for each target in the image data to obtain the target identification results of each target in the image data; According to the target identification results of each target in the image data, mark the main target, auxiliary target, main target key points, and auxiliary target key points in the image data.

5. The method according to claim 3, wherein The target identification of the targets in each of the image data and the marking of the main target, auxiliary target, main target key points, and auxiliary target key points in each of the image data according to the target identification results respectively include: Determine the poses of each of the targets in the lens coordinate systems corresponding to each of the image data respectively; Select one image data from each of the image data to obtain the first image data, where the other image data except the first image data in each of the image data is the second image data; Set unique target identifications for each target in the first image data respectively to obtain the main target and auxiliary target in the first image data; For each second image data, traverse the matching relationship between the targets in the second image data and the targets in the first image data to obtain the relative poses of the camera under each matching relationship; Calculate the errors of each of the relative poses respectively according to the poses of each of the targets in the lens coordinate system corresponding to the second image data and the poses of each of the targets in the lens coordinate system corresponding to the first image data; According to the unique target identifications of the main target and auxiliary target in the first image data, mark the main target, auxiliary target, main target key points, and auxiliary target key points in the second image data according to the matching relationship of the relative pose with the minimum error.

6. The method according to claim 1, wherein Each of the image data is captured by the same lens of the camera at different positions; The calibration of the parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates includes: Calibrate the internal parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates.

7. The method according to claim 1, characterized in that, The at least two image data include the third image data collected by the first lens of the camera at the first position and the fourth image data collected by the second lens of the camera at the second position; The calibration of the parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates includes: Calibrate the transformation parameters of the lens coordinate system of the first lens and the lens coordinate system of the second lens according to the third conversion relationship corresponding to the third image data, the second image coordinates in the third image data, the third conversion relationship corresponding to the fourth image data, the second image coordinates in the fourth image data, and the second target coordinates.

8. A camera calibration device, characterized in that, Include: The first data acquisition module is used to acquire at least two image data of the target array captured by the camera, and acquire the first target coordinates of the main target key points of the main target in the main target coordinate system; wherein, the camera includes at least one lens, and each of the image data is captured by the lens at different positions. The target array includes the main target and at least one auxiliary target, and the poses of the main target and the auxiliary target remain unchanged during the image data capture process; The second data acquisition module is used to, for each auxiliary target, acquire the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system, and acquire the second target coordinates of the auxiliary target key points of the auxiliary target in the auxiliary target coordinate system of the auxiliary target; The key point detection module is used to perform target key point detection on each of the image data, and respectively obtain the first image coordinates of the main target key points and the second image coordinates of the auxiliary target key points in each image data; The first conversion relationship determination module is used to, for each image data, determine the second conversion relationship between the main target coordinate system and the lens coordinate system corresponding to the image data according to the first image coordinates in the image data and the first target coordinates; wherein, the lens coordinate system corresponding to the image data is the lens coordinate system when the camera captures the image data; The second conversion relationship determination module is used to determine the third conversion relationship between the auxiliary target coordinate system and the lens coordinate system corresponding to the image data according to the second conversion relationship and the first conversion relationship; The parameter calibration module is used to calibrate the parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates; 9. The device according to claim 8, characterized in that The second data acquisition module is specifically used for: acquiring the first pose of the auxiliary target in the three-dimensional coordinate system of the shooting scene measured by a high-precision measuring instrument, and acquiring the second pose of the main target in the three-dimensional coordinate system of the shooting scene measured by the high-precision measuring instrument; determining the first conversion relationship between the auxiliary target coordinate system of the auxiliary target and the main target coordinate system according to the first pose and the second pose; The key point detection module includes: a target detection sub-module for performing target detection on each of the image data to respectively obtain each target in each image data; a key point recognition sub-module for performing target key point recognition on each of the targets to respectively obtain the image coordinates of the target key points of each of the targets; a target recognition sub-module for performing target recognition on the targets in each of the image data, and respectively labeling the main target, the auxiliary target, the main target key points and the auxiliary target key points in each of the image data according to the target recognition results, wherein the image coordinates of the main target key points are the first image coordinates, and the image coordinates of the auxiliary target key points are the second image coordinates; A unique target identification pattern is respectively set on the main target and the auxiliary target; the target recognition sub-module is specifically used for: For each piece of image data, identify the unique target identification pattern for each target in the image data to obtain the target identification results of each target in the image data; According to the target identification results of each target in the image data, mark the main target, auxiliary target, main target key points, and auxiliary target key points in the image data; The target identification sub-module is specifically used for: Respectively determine the poses of each of the targets in the lens coordinate system corresponding to each of the image data; Select one piece of image data from each of the image data to obtain first image data, where the other image data except the first image data in each of the image data is second image data; Respectively set unique target identifications for each target in the first image data to obtain the main target and auxiliary target in the first image data; For each second image data, traverse the matching relationship between the targets in the second image data and the targets in the first image data to obtain the relative poses of the camera under each matching relationship; According to the poses of each of the targets in the lens coordinate system corresponding to the second image data and the poses of each of the targets in the lens coordinate system corresponding to the first image data, calculate the errors of each of the relative poses respectively; According to the unique target identifications of the main target and auxiliary target in the first image data, mark the main target, auxiliary target, main target key points, and auxiliary target key points in the second image data according to the matching relationship of the relative pose with the smallest error; Each of the image data is captured by the same lens of the camera at different positions; The parameter calibration module is specifically used for: Calibrate the internal parameters of the camera according to the third conversion relationship corresponding to each of the image data, the second image coordinates in each of the image data, and the second target coordinates; The at least two pieces of image data include third image data collected by the first lens of the camera at the first position and fourth image data collected by the second lens of the camera at the second position; The parameter calibration module is specifically used for: Calibrate the transformation parameters of the lens coordinate system of the first lens and the lens coordinate system of the second lens according to the third conversion relationship corresponding to the third image data, the second image coordinates in the third image data, the third conversion relationship corresponding to the fourth image data, the second image coordinates in the fourth image data, and the second target coordinates.

10. An electronic device, characterized in that, It includes: A memory for storing a computer program; A processor, when executing the program stored on the memory, implements the method according to any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1-7.

Images