A method for extrinsic calibration of multi-view visual inspection system

Abstract

A method for calibrating external parameters of a multi-view visual inspection system belongs to the field of visual inspection technology. The present invention solves the problem that the implementation of a high-precision external parameter calibration method for a multi-view visual inspection system with a low or no common field of view needs to rely on a stereoscopic target with high manufacturing precision. The method of the present invention is as follows: Step 1, according to the working distance of each camera in the multi-view visual inspection system, a plane target is selected for each camera respectively, and the selected plane targets are rigidly combined into a stereoscopic target; Step 2, the plane target is stimulated from each degree of freedom direction, and each camera in the multi-view visual inspection system is used to collect images; Step 3, for each image in each group of images that meets the integrity condition, the pose of the plane target relative to the camera coordinate system when each image is collected is solved respectively; Step 4, the relative pose between any two cameras is solved according to the collected image and the residual function. The method of the present invention can be applied to the field of visual inspection technology.

Description

Technical Field

[0001] The invention belongs to the technical field of visual detection, and in particular relates to an external parameter calibration method for a multi-view visual detection system. Background Art

[0002] In the process of automated assembly of large-aperture optical components, in order to achieve high-precision 6D pose detection of target components, three mutually orthogonal 2D cameras are used to form the hardware system of visual inspection. During the assembly process, the information of different perspectives is aligned to a unified coordinate system through the camera's internal and external calibration data, and the 6D pose of the target component with high resolution in each degree of freedom direction is obtained after calculation. In order to ensure that the pose detection results of the target component have high resolution and high precision attributes, high requirements are placed on vision.

[0003] In the existing multi-camera stereo calibration methods, there are still difficulties in calibrating the external parameters of multi-view visual inspection systems with low overlapping fields of view. Specifically, the planar target-based solution requires multiple cameras to simultaneously capture the target pattern. For multi-view systems with large optical axis angles (especially high-precision measurement systems with small object distances and large apertures), it is easy for the target pattern to be partially out of focus. In order to achieve high-precision external parameter calibration of stereo targets, high manufacturing precision is required, which will bring extremely high costs.

[0004] In summary, the existing external parameter calibration method requires a three-dimensional target with very high manufacturing precision to be implemented. Therefore, it is necessary to propose a new external parameter calibration method to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the implementation of a high-precision external parameter calibration method for a multi-view visual inspection system with low or no common field of view needs to rely on a stereo target with high manufacturing precision, and to propose an external parameter calibration method for a multi-view visual inspection system.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a method for calibrating external parameters of a multi-view visual inspection system, the method specifically comprising the following steps:

[0007] Step 1: According to the working distance of each camera in the multi-view visual inspection system, a plane target is selected for each camera, and then the selected plane targets are rigidly combined into a three-dimensional target;

[0008] Step 2: Stimulate the planar target from each degree of freedom direction, and use each camera in the multi-view visual inspection system to collect images, and take the images collected simultaneously by the multi-view visual inspection system as a group;

[0009] Step 3: For each image in any group of images that meets the integrity condition, respectively calculate the pose of the plane target relative to the camera coordinate system when each image is acquired;

[0010] Similarly, each image in each group of images that meets the integrity condition is processed;

[0011] Step 4: Construct a residual function and solve the relative pose between any two cameras and the relative pose between any two planar targets based on the collected images and the residual function.

[0012] Furthermore, the selected planar targets are rigidly combined into three-dimensional targets; specifically:

[0013] According to the positional relationship of the detection space of each camera, the plane targets corresponding to each camera are rigidly combined into a three-dimensional target.

[0014] Furthermore, the multi-view visual inspection system is a three-view visual inspection system, that is, the system includes 1 main camera and 2 side cameras.

[0015] Furthermore, when collecting images, at least two cameras are required to meet the integrity condition;

[0016] The integrity condition is: the number of complete April tags included in the image captured by the current camera is recorded as m, and the number of April tags contained in the plane target corresponding to the current camera is recorded as n. When the ratio of m to n is greater than or equal to 25%, it is considered that the image captured by the current camera meets the integrity condition.

[0017] Furthermore, the pose of the planar target relative to the camera coordinate system when each image is captured is solved separately; specifically:

[0018] Define the lower left corner of Apriltag with id=0 as the origin of the plane target coordinate system, the direction from the lower left corner to the lower right corner is the positive direction of the x-axis, and the direction from the lower left corner to the upper left corner is the positive direction of the y-axis;

[0019] For any image that meets the integrity condition, the coordinates of the lower left corner, lower right corner, upper right corner, and upper left corner of Apriltag in the image in the plane target coordinate system are ((a+c)*id%l,(a+c)*id / l,0), ((a+c)*id%l+a,(a+c)*id / l,0), ((a+c)*id%l+a,(a+c)*id / l+a,0), ((a+c)*id%l,(a+c)*id / l+a,0);

[0020] Where a is the side length of a single April tag, c is the distance between adjacent April tags, l is the number of April tags in each row of the plane target, % is the remainder operation, and / is the integer division operation;

[0021] According to the coordinates of the lower left corner, lower right corner, upper right corner and upper left corner of Apriltag in the image in the image coordinate system, the coordinates in the plane target coordinate system and the internal parameters of the camera, the pose of the plane target corresponding to the image relative to the camera coordinate system corresponding to the image is calculated.

[0022] Furthermore, the pose of the planar target corresponding to the image relative to the camera coordinate system corresponding to the image is calculated using a PNP algorithm.

[0023] Furthermore, the specific process of step 4 is as follows:

[0024] Step 41: For cameras C1 and C2 in the multi-view visual inspection system, in the kth group of images collected, the homogeneous matrix of the pose of the plane target corresponding to camera C1 in the camera C1 coordinate system is recorded as A k , the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the camera C2 coordinate system is recorded as B k ; k = 1, 2, ..., K, K is the number of image groups of cameras C1 and C2 that simultaneously meet the integrity condition;

[0025] Step 42: Let the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera C1 be X, and let the homogeneous matrix of the pose of camera C2 in the coordinate system of camera C1 be Y, then A k X=YB k ;

[0026] Step 43: Construct the residual function E:

[0027] E=λE P +(1-λ)E rj

[0028] Among them, E P is an intermediate variable, E rj is the reprojection error, λ is the weight;

[0029]

[0030] Among them, Dis(A k X-YB k ) represents the homogeneous matrix A k X and YB k Frobenius norm of the rotation matrix difference and homogeneous matrix A k X and YB kThe weighted sum of the magnitudes of the translation vector differences in ;

[0031] The coordinates of the corner point on the plane target corresponding to camera C1 in the plane target coordinate system are expressed as P i , P i The coordinates of the kth group of images captured by camera C1 in the image coordinate system are p i ,but

[0032]

[0033]

[0034] Among them, K A is the intrinsic parameter matrix of camera C1, which will calculate p i The process is recorded as

[0035] The coordinates of the corner point on the plane target corresponding to camera C2 in the plane target coordinate system are expressed as P′ i′ , P′ i′ The coordinates of the kth group of images captured by camera C2 in the image coordinate system are p′ i′ ,but

[0036]

[0037]

[0038] Among them, K B is the intrinsic parameter matrix of camera C2, which will calculate p′ i′ The process is recorded as

[0039]

[0040]

[0041] Wherein, Norm represents the calculation of the Euclidean distance, I is the number of corner points corresponding to the plane target contained in the k-th group of images acquired by camera C1, and I′ is the number of corner points corresponding to the plane target contained in the k-th group of images acquired by camera C2. YesP i The actual coordinates corresponding to the image coordinate system collected by camera C1, is P′ i′ The actual coordinates corresponding to the image coordinate system collected by camera C2;

[0042] The reprojection error is E rj :

[0043]

[0044] Step 44, solve the homogeneous matrices X and Y by minimizing the residual function value;

[0045] Step 45: For every two cameras in the multi-view visual inspection system, the process from step 41 to step 44 is performed.

[0046] Furthermore, the homogeneous matrices X and Y are calculated using a Levenberg-Marquadt algorithm.

[0047] Furthermore, the method further includes step five, which is specifically:

[0048] The pose of camera C1 in the camera G coordinate system is recorded as Y1, the pose of camera C2 in the camera G coordinate system is recorded as Y2, and the pose of camera C2 in the camera C1 coordinate system is recorded as Y3.

[0049] Y1 -1 Y2=Y3

[0050] Among them, the superscript -1 represents the inverse of the matrix;

[0051] The homogeneous matrix of the pose of the plane target corresponding to camera C1 in the plane target coordinate system corresponding to camera G is recorded as X1, the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera G is recorded as X2, and the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera C1 is recorded as X3, then

[0052] X1 -1 X2=X3

[0053] The homogeneous matrix of the pose of the plane target corresponding to camera G in the camera G coordinate system is recorded as C k' , the homogeneous matrix of the pose of the plane target corresponding to camera C1 in the camera C1 coordinate system is recorded as A k' , the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the camera C2 coordinate system is recorded as B k' ; k'=1,2,…,K', K' is the number of image groups that meet the integrity condition of the three cameras at the same time;

[0054]

[0055] Among them, E′ p is an intermediate variable;

[0056] For the k'th group of images, the reprojection errors of camera G, camera C1 and camera C2 are:

[0057]

[0058]

[0059]

[0060] in, is the reprojection error of camera G, is the reprojection error of camera C1, is the reprojection error of camera C2, K G is the intrinsic parameter matrix of camera G, P l is the coordinate of the corner point on the plane target corresponding to the camera G in the plane target coordinate system, and They are P l The coordinate calculation values ​​in the image coordinate system of the k'th group of images collected by camera G are reprojected in two different ways, YesP l The actual coordinates corresponding to the image coordinate system of the image captured by camera G, L is the number of corner points corresponding to the plane target detected in the k'th group of images captured by camera G, P l′ is the coordinate of the corner point on the plane target corresponding to camera C1 in the plane target coordinate system, and They are P l′ The coordinate calculation values ​​in the image coordinate system of the k'th group of images acquired by camera C1 are reprojected in two different ways, YesP l′ The actual coordinates corresponding to the image coordinate system of the image captured by camera C1, L′ is the number of corner points corresponding to the planar target detected in the k′th group of images captured by camera C1, P l″ is the coordinate of the corner point on the plane target corresponding to camera C2 in the plane target coordinate system, and They are P l″ The coordinate calculation values ​​in the image coordinate system of the k'th group of images acquired by camera C2 are reprojected in two different ways, YesP l″ The actual coordinates corresponding to the image coordinate system of the image captured by camera C2, L″ is the number of corner points corresponding to the plane target detected in the k'th group of images captured by camera C2;

[0061] The overall reprojection error is then:

[0062]

[0063] The constructed residual function is:

[0064]

[0065] Among them, λ' is the weight coefficient;

[0066] Take the relative pose result calculated in step 4 as the initial value, use the Levenberg-Marquadt algorithm to minimize the value of the residual function E′, and solve the homogeneous matrices X1, X2, X3 and Y1, Y2, Y3;

[0067] Taking the camera G coordinate system as the reference coordinate system, the homogeneous matrix of the external parameters of camera G is E 4*4 , E 4*4 is the identity matrix, the homogeneous matrix of the external parameters of camera C1 is Y1, and the homogeneous matrix of the external parameters of camera C2 is Y2.

[0068] The beneficial effects of the present invention are:

[0069] (1) Low cost and high precision: Compared with the traditional method that requires high-precision processing of three-dimensional targets, the present invention greatly reduces the cost by using a three-dimensional target formed by a rigid combination of multiple planar targets. The method of the present invention does not need to ensure the precise spatial position relationship between the targets, but only needs to maintain the rigidity between the targets to meet the requirements of high-precision calibration. At the same time, it solves the problem that although the traditional planar target has a low cost, it is not ideal in a small depth of field measurement system or a system with a low common view field of view.

[0070] (2) Flexible target combination: When combining planar targets to form three-dimensional targets, it is not necessary to predetermine the precise spatial position relationship between the targets, but only to ensure the rigidity between them. This flexibility allows users to freely combine the required targets according to specific needs and site conditions, providing great adaptability and convenience.

[0071] (3) Simple and easy-to-use external parameter solution algorithm: The algorithm proposed in the present invention does not require complex CAD data or use it as parameter input, but directly uses the parameters of the planar target for calculation. This method simplifies the entire calibration process, making the operation more concise, easy to understand and implement.

[0072] (4) High-precision calibration results: The calibration algorithm of the present invention integrates multiple physical constraints and parameters to improve the interpretability of the solution. Even when using a low-cost combined stereo target, it can achieve a calibration accuracy comparable to that of a high-precision stereo target, thereby ensuring the accuracy and reliability of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a flow chart of an external parameter calibration method of a multi-view visual inspection system of the present invention;

[0074] Figure 2 It is a schematic diagram of an orthogonal three-view visual inspection system;

[0075] Figure 3Schematic diagram of the Aprilgrid calibration plate for the main camera;

[0076] Figure 4 Schematic diagram of a combined three-dimensional target;

[0077] Figure 5 This is a schematic diagram of the system when collecting images;

[0078] Figure 6 Schematic diagram of the process of calculating homogeneous matrices X and Y. DETAILED DESCRIPTION

[0079] In general, the present invention reduces costs while ensuring high accuracy of calibration results through innovative target design, flexible combination methods, simplified solution algorithms and strict requirements on accuracy, and adapts to various complex multi-view visual inspection system requirements.

[0080] Specific implementation method 1: Combination Figure 1 This embodiment describes an external parameter calibration method for a multi-view visual inspection system, and the method specifically includes the following steps:

[0081] Step 1: According to the working distance of each camera in the multi-view visual inspection system, a plane target is selected for each camera, and then the selected plane targets are rigidly combined into a three-dimensional target;

[0082] Step 2: Stimulate the planar target from each degree of freedom direction, and use each camera in the multi-view visual inspection system to collect images, and use the images collected by the multi-view visual inspection system at the same time as a group (when using the multi-view visual inspection system to collect images, each camera collects an image during each collection process, and the images collected by all cameras are used as a group of images collected at the current time);

[0083] Step 3: for each image in any group of images that meets the integrity condition, respectively calculate the pose of the planar target relative to the camera coordinate system when each image (i.e., the image in the group of images that meets the integrity condition) is acquired;

[0084] Similarly, each image in each group of images that meets the integrity condition is processed;

[0085] Step 4: Construct a residual function and solve the relative pose between any two cameras and the relative pose between any two planar targets based on the collected images and the residual function.

[0086] After obtaining the relative pose between any two cameras, the homogeneous matrix of the external parameters of each camera is obtained.

[0087] The present invention provides a high-precision external parameter calibration method for a multi-view visual inspection system with a low overlapping field of view. According to the design of the visual inspection system, a plurality of planar high-precision targets are rigidly combined to form a new three-dimensional target, and a matching calibration algorithm is used. There is no need to know the spatial position relationship between the plurality of planar targets, so as to achieve high-precision external parameter calibration of the multi-view visual inspection system. Using the calculated external parameter data and the known camera internal parameter data, the 2D information collected by different cameras can be uniformly aligned to the 3D global coordinate system to achieve higher spatial resolution and precision measurement. By improving the position accuracy of optical component detection, the needs of high-precision assembly and detection can be met.

[0088] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that the selected planar targets are rigidly combined into three-dimensional targets; specifically:

[0089] According to the positional relationship of the detection space of each camera, the plane targets corresponding to each camera are rigidly combined into a three-dimensional target.

[0090] The other steps and parameters are the same as those in the first embodiment.

[0091] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that: the multi-view visual detection system is a three-view visual detection system, that is, the system includes 1 main camera (denoted as camera G) and 2 side cameras (denoted as camera C1 and camera C2).

[0092] The other steps and parameters are the same as those in the first or second embodiment.

[0093] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that: when collecting images, at least two cameras are required to meet the integrity condition;

[0094] The integrity condition is: the number of complete April tags included in the image captured by the current camera is recorded as m, and the number of April tags contained in the plane target corresponding to the current camera is recorded as n. When the ratio of m to n is greater than or equal to 25%, it is considered that the image captured by the current camera meets the integrity condition.

[0095] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0096] Specific implementation method 5: This implementation method is different from any one of the specific implementation methods 1 to 4 in that: the pose of the plane target relative to the camera coordinate system when each image is captured is solved separately; specifically:

[0097] Define the lower left corner of Apriltag with id=0 as the origin of the plane target coordinate system, the direction from the lower left corner to the lower right corner is the positive direction of the x-axis, and the direction from the lower left corner to the upper left corner is the positive direction of the y-axis;

[0098] For any image that meets the integrity condition, the coordinates of the lower left corner, lower right corner, upper right corner, and upper left corner of Apriltag in the image in the plane target coordinate system are ((a+c)*id%l,(a+c)*id / l,0), ((a+c)*id%l+a,(a+c)*id / l,0), ((a+c)*id%l+a,(a+c)*id / l+a,0), ((a+c)*id%l,(a+c)*id / l+a,0);

[0099] Where a is the side length of a single April tag, c is the distance between adjacent April tags, l is the number of April tags in each row of the plane target, % is the remainder operation, and / is the integer division operation;

[0100] According to the coordinates of the lower left corner, lower right corner, upper right corner and upper left corner of Apriltag in the image in the image coordinate system, the coordinates in the plane target coordinate system and the internal parameters of the camera, the pose of the plane target corresponding to the image relative to the camera coordinate system corresponding to the image is calculated.

[0101] Using the AprilTag algorithm, in each image that meets the integrity condition, detect the ID of the completely collected Apriltag and the coordinates of the completely collected Apriltag corner points in the image coordinate system, establish a correspondence between the image coordinates of all detected Apriltag corner points and their coordinates in the calibration plate coordinate system, and use the PNP algorithm based on the internal parameters of the camera and the image coordinates of the detected corner points and the coordinates in the calibration plate coordinate system to solve the position of the calibration plate relative to the camera coordinate system when collecting images.

[0102] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0103] Specific implementation method six: This implementation method is different from any one of specific implementation methods one to five in that: the pose of the planar target corresponding to the image relative to the camera coordinate system corresponding to the image is solved by using the PNP (Perspective-n-Point) algorithm.

[0104] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0105] Specific implementation method 7: This implementation method is different from any one of the specific implementation methods 1 to 6 in that the specific process of step 4 is:

[0106] Step 41: For cameras C1 and C2 in the multi-view visual inspection system, in the kth group of images collected, the homogeneous matrix of the pose of the plane target corresponding to camera C1 in the camera C1 coordinate system is recorded as A k , the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the camera C2 coordinate system is recorded as B k ; k = 1, 2, ..., K, K is the number of image groups of cameras C1 and C2 that simultaneously meet the integrity condition;

[0107] Homogeneous matrix A k and the homogeneous matrix B k Obtained according to the pose of the planar target relative to the camera coordinate system;

[0108] Step 42: Let the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera C1 be X, and let the homogeneous matrix of the pose of camera C2 in the coordinate system of camera C1 be Y, then A k X=YB k ;

[0109] Step 43: Construct the residual function E:

[0110] E=λE P +(1-λ)E rj

[0111] Among them, E P is an intermediate variable, E rj is the reprojection error, λ is the weight;

[0112]

[0113] Among them, Dis(A k X-YB k ) represents the homogeneous matrix A k X and YB k Frobenius norm of the rotation matrix difference and homogeneous matrix A k X and YB k The weighted sum of the magnitudes of the translation vector differences in ;

[0114] The coordinates of the corner point on the plane target corresponding to camera C1 in the plane target coordinate system are expressed as P i , P i The coordinates of the kth group of images captured by camera C1 in the image coordinate system are p i ,but

[0115]

[0116]

[0117] Among them, K A is the intrinsic parameter matrix of camera C1, which will calculate p i The process is recorded as

[0118] The coordinates of the corner point on the plane target corresponding to camera C2 in the plane target coordinate system are expressed as P′ i′ , P′ i′ The coordinates of the kth group of images captured by camera C2 in the image coordinate system are p′ i′ ,but

[0119]

[0120]

[0121] Among them, K B is the intrinsic parameter matrix of camera C2, which will calculate p i The process of "

[0122]

[0123]

[0124] Wherein, Norm represents the calculation of the Euclidean distance, I is the number of corner points corresponding to the plane target contained in the k-th group of images acquired by camera C1 (that is, find the image acquired by camera C1 in the k-th group of images acquired, and then determine the number of corner points corresponding to the plane target contained in the image acquired by camera C1), I′ is the number of corner points corresponding to the plane target contained in the k-th group of images acquired by camera C2, YesP i The actual coordinates corresponding to the image coordinate system collected by camera C1, is P′ i′ The actual coordinates corresponding to the image coordinate system collected by camera C2;

[0125] The reprojection error is E rj :

[0126]

[0127] Step 44, solve the homogeneous matrices X and Y by minimizing the residual function value;

[0128] Step 45: For every two cameras in the multi-view visual inspection system, the process from step 41 to step 44 is performed.

[0129] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0130] The following is Dis(A kX-YB k ) operation is explained as follows:

[0131] First, calculate the homogeneous matrix A k Rotation matrices in X and YB k The difference of the rotation matrices in , and then calculate the Frobenius norm of the difference;

[0132] Next, calculate the homogeneous matrix A k Translation vector in X and YB k The difference of the translation vectors in , and then calculate the modulus of the difference;

[0133] Finally, the weighted sum of the Frobenius norm value and the modulus value is calculated. In the present invention, the ratio of the weight of the Frobenius norm value and the weight of the modulus value is set to 10:1.

[0134] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that: the homogeneous matrices X and Y are calculated using the Levenberg-Marquadt algorithm.

[0135] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0136] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that: the method further includes step 5, which is specifically:

[0137] The pose of camera C1 in the camera G coordinate system is recorded as Y1, the pose of camera C2 in the camera G coordinate system is recorded as Y2, and the pose of camera C2 in the camera C1 coordinate system is recorded as Y3.

[0138] Y1 -1 Y2=Y3

[0139] Among them, the superscript -1 represents the inverse of the matrix;

[0140] The homogeneous matrix of the pose of the plane target corresponding to camera C1 in the plane target coordinate system corresponding to camera G is recorded as X1, the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera G is recorded as X2, and the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera C1 is recorded as X3, then

[0141] X1 -1 X2=X3

[0142] The homogeneous matrix of the pose of the plane target corresponding to camera G in the camera G coordinate system is recorded as C k' , the homogeneous matrix of the pose of the plane target corresponding to camera C1 in the camera C1 coordinate system is recorded as A k', the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the camera C2 coordinate system is recorded as B k' ; k'=1,2,…,K', K' is the number of image groups that meet the integrity condition of the three cameras at the same time;

[0143]

[0144] Among them, E p ′ is an intermediate variable;

[0145] For the k'th group of images, the reprojection errors of camera G, camera C1 and camera C2 are:

[0146]

[0147]

[0148]

[0149] in, is the reprojection error of camera G, is the reprojection error of camera C1, is the reprojection error of camera C2, K G is the intrinsic parameter matrix of camera G, P l is the coordinate of the corner point on the plane target corresponding to the camera G in the plane target coordinate system, and They are P l The coordinate calculation values ​​in the image coordinate system of the k'th group of images collected by camera G are reprojected in two different ways, YesP l The actual coordinates corresponding to the image coordinate system of the image captured by camera G, L is the number of corner points corresponding to the plane target detected in the k'th group of images captured by camera G, P l′ is the coordinate of the corner point on the plane target corresponding to camera C1 in the plane target coordinate system, and They are P l′ The coordinate calculation values ​​in the image coordinate system of the k'th group of images acquired by camera C1 are reprojected in two different ways, YesP l′ The actual coordinates corresponding to the image coordinate system of the image captured by camera C1, L′ is the number of corner points corresponding to the planar target detected in the k′th group of images captured by camera C1, P l″ is the coordinate of the corner point on the plane target corresponding to camera C2 in the plane target coordinate system, and They are P l″The coordinate calculation values ​​in the image coordinate system of the k'th group of images acquired by camera C2 are reprojected in two different ways, YesP l″ The actual coordinates corresponding to the image coordinate system of the image captured by camera C2, L″ is the number of corner points corresponding to the plane target detected in the k'th group of images captured by camera C2;

[0150] The overall reprojection error is then:

[0151]

[0152] The constructed residual function is:

[0153] E′=λ'E′ P +(1-λ')E′ rj

[0154] Among them, λ' is the weight coefficient;

[0155] Take the relative pose result calculated in step 4 as the initial value, use the Levenberg-Marquadt algorithm to minimize the value of the residual function E′, and solve the homogeneous matrices X1, X2, X3 and Y1, Y2, Y3.

[0156] Taking the camera G coordinate system as the reference coordinate system, the homogeneous matrix of the external parameters of camera G is E 4*4 , E 4*4 is the identity matrix, the homogeneous matrix of the external parameters of camera C1 is Y1, and the homogeneous matrix of the external parameters of camera C2 is Y2.

[0157] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0158] Example

[0159] Figure 2 Schematic diagram of an orthogonal three-view visual inspection system for high-precision posture detection of large-size components in industrial automation, where camera G is the main camera of the orthogonal three-view visual inspection system and the origin of the main camera coordinate system is O G , the main camera coordinate system is used as the global coordinate system, cameras C1 and C2 are the side cameras of the orthogonal three-view visual inspection system, and the origin of the side camera C1 coordinate system and the origin of the side camera C2 coordinate system are O C1 and O C2 In the external parameter calibration, the 6D pose of the two side cameras relative to the main camera needs to be measured. The external parameters of the system are calibrated using the method of this embodiment.

[0160] Step 1: Assemble the modular 3D target

[0161] According to the working distance, field of view and other parameters of each camera in the orthogonal three-view visual inspection system, select a suitable plane target: the working distance of the main camera is 1200mm, and the actual size of the field of view at the working distance is 600mm*600mm. The working distance of the two side-view cameras is 400mm, and the actual size of the field of view at the working distance is 300mm*150mm. The size of the selected plane target should be smaller than the actual size of the field of view. In order to ensure that the calibration function can be normally realized even if no more than 75% of the target is not in the camera's field of view, the Aprilgrid calibration plate composed of the Apriltag matrix is ​​used as the plane target. The calibration plate is a customized alumina glass substrate calibration plate with a pattern accuracy of ±0.01mm. For the main camera, the Aprilgrid calibration plate contains 6*6 (r*l) Apriltags with ids ranging from 0 to 35. The side length a of a single Apriltag is 60mm, and the side length c of the separating pattern is 6mm. The schematic diagram of the Aprilgrid calibration plate of the main camera is shown in the figure below. Figure 3 For each side-view camera, the Aprilgrid calibration plate contains 3*6 (r*l) Apriltags with ids from 0 to 17. The side length a of a single Apriltag is 60 mm, and the side length c of the separation pattern is 6 mm.

[0162] According to the spatial position relationship of the detection space of each camera, the corresponding plane targets are rigidly combined into a three-dimensional target suitable for the orthogonal three-view visual inspection system: select the support for fixing the three plane targets. Since the three cameras of the orthogonal three-view visual inspection system are orthogonal to each other, according to the approximate spatial position relationship of the effective area detected by the three cameras, and considering the size of the calibration plate, an ABS cube with a size of 450mm*450mm*250mm is selected as the base of the combined three-dimensional target. The selected plane target is rigidly fixed to the base with curing glue, which is the required combined three-dimensional target. The schematic diagram of the combined three-dimensional target is shown in the figure. Figure 4 As shown, the combined stereo target has at least a series of postures in three-dimensional space, so that each camera can simultaneously clearly and completely image its corresponding plane target.

[0163] Step 2: Calibration Image Acquisition

[0164] In order to accurately calculate the external parameters of the orthogonal three-view visual inspection system, it is necessary to stimulate the planar target from all degrees of freedom, such as Figure 5As shown, the images containing the constructed stereo targets are collected multiple times at the same time. Each time the orthogonal three-view visual inspection system collects images simultaneously, it is a group. During the collection, in order to ensure the calibration accuracy, the number of complete April tags captured by the camera in the corresponding calibration plate is not less than 25% of the number of April tags contained in the calibration plate as the integrity condition, and at least two cameras should be guaranteed to meet the integrity condition. Under the premise of meeting the above requirements, the stereo target is moved evenly in the 6 degrees of freedom directions, and a group of calibration images is collected after each movement until the system is stable. According to the situation that the integrity condition is met in each group of calibration images, the calibration images of all groups are divided into 4 non-mutually exclusive categories: the main camera and the side camera C1 meet the integrity condition, the main camera and the side camera C2 meet the integrity condition, the side camera C1 and the side camera C2 meet the integrity condition, and all cameras meet the integrity condition. It is required that the number of image groups contained in each category is not less than 12 groups.

[0165] Step 3: Solve for camera extrinsics

[0166] Assume that the internal parameters of all cameras have been calibrated at this time.

[0167] a. Rough solution of the calibration plate’s pose relative to the camera.

[0168] In order to obtain the external parameters of the visual inspection system, that is, the position vector (x, y, z) and rotation matrix R of each camera in the visual inspection system relative to the global coordinate system, firstly, the pose of the corresponding calibration plate in the respective camera coordinate system during shooting is solved from the acquired image. A lookup table for each calibration plate at the real scale is constructed. Define the lower left corner of Apriltag with id=0 as the origin of the calibration plate coordinate system, define the direction from the lower left corner to the lower right corner as the positive direction of the x-axis, and define the direction from the lower left corner to the upper left corner as the positive direction of the y-axis. Then the coordinates of the lower left, lower right, upper right, and upper left corners of the detected Apriltag in the calibration plate coordinate system are ((a+c)*id%l,(a+c)*id / l,0), ((a+c)*id%l+a,(a+c)*id / l,0), ((a+c)*id%l+a,(a+c)*id / l+a,0), ((a+c)*id%l,(a+c)*id / l+a,0). Where % is the remainder operation, / is the integer division operation, l is the number of Apriltags contained in each row of the plane target, and r is the number of Apriltags contained in each column of the plane target.

[0169] Using the AprilTag algorithm, in each image that meets the integrity conditions, the ID of the completely collected Apriltag and the coordinates of the corner points in the image coordinate system are detected, and a corresponding relationship is established between the image coordinates of all detected Apriltag corner points and their coordinates in the calibration plate coordinate system. Based on the internal parameters of the camera, the image coordinates of the detected corner points, and the coordinates in the calibration plate coordinate system, the PNP algorithm is used to solve the position of the calibration plate relative to the camera coordinate system when the image is collected.

[0170] b. By constructing and minimizing the residual function, the 6D pose between two cameras is roughly calculated.

[0171] The residual function consists of two parts: the difference E between two equivalent paths of the closed pose transfer chain p And the reprojection error E rj .

[0172] For cameras C1 and C2, assume that the 1st, 2nd, ..., Kth groups of images (in order to ensure the accuracy of the calibration results, K>12 is required, which does not mean that the method fails when K is a smaller value) are all the images collected by the two cameras and meet the integrity requirements. At this time, it can be known from step a that in the kth group of images, the homogeneous matrix of the corresponding plane target in the respective camera coordinate system is A k , B k , let X be the homogeneous matrix of the position of the calibration plate corresponding to camera C2 in the coordinate system of the calibration plate corresponding to camera C1, and Y be the homogeneous matrix of the position of camera C2 in the coordinate system of camera C1. k and B k , since X and Y do not change with the rigid motion of the stereo target, as different representations of the position and posture of camera C1 relative to the calibration plate corresponding to camera C2, the second matrix representing the above position and posture corresponding to the kth observation (k = 1, 2, ..., K) satisfies A k X=YB k . Then the residual function formed by the difference between the two equivalent paths of the closed posture transfer chain is:

[0173]

[0174] Assume that the coordinates of the marked point on a calibration plate in the calibration plate coordinate system are P, the coordinates of the marked point in the image coordinate system during observation are p, and the intrinsic parameter matrix of the camera is K. If A k is unbiased, then:

[0175]

[0176]

[0177] This process is recorded as p = π K (A k ,P), assuming that I corner points corresponding to the calibration plate are detected in the k-th image of camera C1, and assuming that I′ corner points corresponding to the calibration plate are detected in the k-th image of camera C2, the reprojection error in this group of images is:

[0178]

[0179]

[0180] Where Norm(p1,p2) represents the Euclidean distance between 2D points p1 and p2, so the reprojection error for the 1st, 2nd, ..., Kth group of images is defined as:

[0181]

[0182] The constructed residual function is:

[0183] E=λE P +(1-λ)E rj

[0184] The Levenberg-Marquadt algorithm is used to minimize the residual function value to solve the homogeneous matrices X and Y. The schematic diagram of the calculation process is as follows Figure 6 As shown in Figure 2, the above method is used to solve the position and posture relationship between the two cameras and the two planar targets for different groups of calibration image data.

[0185] For some application scenarios, the current camera extrinsic parameters can already meet the accuracy requirements. If a higher-precision extrinsic parameter calibration result is required, a subsequent extrinsic parameter refinement step c is required.

[0186] c. Refine the camera extrinsic data.

[0187] In order to achieve a more refined external parameter calibration result, more physical constraints are used to refine the calibration result obtained in step b. The physical constraints are pose transfer closed loop constraints and reprojection constraints. In step b, the 6D pose relationship between the three cameras and the 6D pose relationship between the three planar targets have been obtained. The pose of camera C1 in the camera G coordinate system is Y1, the pose of camera C2 in the camera G coordinate system is Y2, and the pose of camera C2 in the camera C1 coordinate system is Y3. Ideally, there should be:

[0188] Y1 -1 Y2=Y3

[0189] Similarly:

[0190] X1 -1 X2=X3

[0191] For all calibration image groups that meet the requirements of all camera integrity, assuming there are K' groups, in the k'th group of images, the homogeneous matrix of the pose of the corresponding planar target in the respective camera coordinate system is C for camera G. k' , corresponding to camera C1 is A k' , corresponding to camera C2 is B k' In this step

[0192]

[0193] For the k'th group of images, the reprojection errors of cameras G, C1, and C2 are:

[0194]

[0195]

[0196]

[0197] The overall reprojection error is:

[0198]

[0199] The residual function constructed in this step is:

[0200] E′=λ'E′ P +(1-λ')E′ rj

[0201] The corresponding values ​​solved in step b are used as initial values, and the homogeneous matrices X1, X2, X3 and Y1, Y2, Y3 are numerically solved by minimizing the residual function using the Levenberg-Marquadt algorithm.

[0202] If the coordinate system of camera G is used as the reference coordinate system, the homogeneous matrix of the external parameters of camera G is E 4*4 , the homogeneous matrix of the external parameters of camera C1 is Y1, and the homogeneous matrix of the external parameters of camera C2 is Y2.

[0203] This embodiment achieves high-precision external parameter calibration of low-overlapping field of view multi-view visual inspection systems through a unique combined stereo target design and precise image acquisition and solution process, improving the problem that such visual inspection systems must use expensive stereo targets in the past, and significantly reducing costs in high-precision external parameter calibration of low-overlapping field of view multi-view visual inspection systems.

[0204] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for calibrating external parameters of a multi-view visual inspection system, characterized in that: The method specifically comprises the following steps: Step 1: According to the working distance of each camera in the multi-view visual inspection system, a plane target is selected for each camera, and then the selected plane targets are rigidly combined into a three-dimensional target; Step 2: Stimulate the planar target from each degree of freedom direction, and use each camera in the multi-view visual inspection system to collect images, and take the images collected simultaneously by the multi-view visual inspection system as a group; Step 3: For each image in any group of images that meets the integrity condition, respectively calculate the pose of the plane target relative to the camera coordinate system when each image is acquired; Similarly, each image in each group of images that meets the integrity condition is processed; Step 4: Construct a residual function, and solve the relative pose between any two cameras and the relative pose between any two planar targets according to the collected images and the residual function; The specific process of step 4 is as follows: Step 41: For cameras C1 and C2 in the multi-view visual inspection system, in the kth group of images collected, the homogeneous matrix of the pose of the plane target corresponding to camera C1 in the camera C1 coordinate system is recorded as A k , the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the camera C2 coordinate system is recorded as B k ; k = 1, 2, ..., K, K is the number of image groups of cameras C1 and C2 that simultaneously meet the integrity condition; Step 42: Let the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera C1 be X, and let the homogeneous matrix of the pose of camera C2 in the coordinate system of camera C1 be Y, then A k X=YB k ; Step 43: Construct the residual function E: E=λE P +(1-λ)E rj Among them, E P is an intermediate variable, E rj is the reprojection error, λ is the weight; Among them, Dis(A k X-YB k ) represents the homogeneous matrix A k X and YB k Frobenius norm of the rotation matrix difference and homogeneous matrix A k X and YB k The weighted sum of the magnitudes of the translation vector differences in ; The coordinates of the corner point on the plane target corresponding to camera C1 in the plane target coordinate system are expressed as P i , P i The coordinates of the kth group of images captured by camera C1 in the image coordinate system are p i ,but Among them, K A is the intrinsic parameter matrix of camera C1, which will calculate p i The process is recorded as The coordinates of the corner points on the plane target corresponding to camera C2 in the plane target coordinate system are expressed as The coordinates of the kth group of images captured by camera C2 in the image coordinate system are but Among them, K B is the intrinsic parameter matrix of camera C2, which will be calculated The process is recorded as Wherein, Norm represents the calculation of the Euclidean distance, I is the number of corner points corresponding to the plane target contained in the k-th group of images acquired by camera C1, and I′ is the number of corner points corresponding to the plane target contained in the k-th group of images acquired by camera C2. YesP i The actual coordinates corresponding to the image coordinate system collected by camera C1, yes The actual coordinates corresponding to the image coordinate system collected by camera C2; The reprojection error is E rj : Step 44, solve the homogeneous matrices X and Y by minimizing the residual function value; Step 45: For every two cameras in the multi-view visual inspection system, the process from step 41 to step 44 is performed.

2. The method for calibrating external parameters of a multi-view visual inspection system according to claim 1, characterized in that: The selected planar targets are rigidly combined into three-dimensional targets; specifically: According to the positional relationship of the detection space of each camera, the plane targets corresponding to each camera are rigidly combined into a three-dimensional target.

3. The method for calibrating external parameters of a multi-view visual inspection system according to claim 2, characterized in that: The multi-view visual inspection system is a three-view visual inspection system, that is, the system includes 1 main camera and 2 side cameras.

4. The method for calibrating external parameters of a multi-view visual inspection system according to claim 3, characterized in that: When collecting images, at least two cameras need to meet the integrity condition; The integrity condition is: the number of complete April tags included in the image captured by the current camera is recorded as m, and the number of April tags contained in the plane target corresponding to the current camera is recorded as n. When the ratio of m to n is greater than or equal to 25%, it is considered that the image captured by the current camera meets the integrity condition.

5. The method for calibrating external parameters of a multi-view visual inspection system according to claim 4, characterized in that: The pose of the planar target relative to the camera coordinate system when each image is collected is solved separately; specifically: Define the lower left corner of Apriltag with id=0 as the origin of the plane target coordinate system, the direction from the lower left corner to the lower right corner is the positive direction of the x-axis, and the direction from the lower left corner to the upper left corner is the positive direction of the y-axis; For any image that meets the integrity condition, the coordinates of the lower left corner, lower right corner, upper right corner, and upper left corner of Apriltag in the image in the plane target coordinate system are ((a+c)*id%l,(a+c)*id / l,0), ((a+c)*id%l+a,(a+c)*id / l,0), ((a+c)*id%l+a,(a+c)*id / l+a,0), ((a+c)*id%l,(a+c)*id / l+a,0); Where a is the side length of a single April tag, c is the distance between adjacent April tags, l is the number of April tags in each row of the plane target, % is the remainder operation, and / is the integer division operation; According to the coordinates of the lower left corner, lower right corner, upper right corner and upper left corner of Apriltag in the image in the image coordinate system, the coordinates in the plane target coordinate system and the internal parameters of the camera, the pose of the plane target corresponding to the image relative to the camera coordinate system corresponding to the image is calculated.

6. The method for calibrating external parameters of a multi-view visual inspection system according to claim 5, characterized in that: The PNP algorithm is used to calculate the pose of the planar target corresponding to the image relative to the camera coordinate system corresponding to the image.

7. The method for calibrating external parameters of a multi-view visual inspection system according to claim 6, characterized in that: The homogeneous matrices X and Y are calculated using the Levenberg-Marquadt algorithm.

8. The method for calibrating external parameters of a multi-view visual inspection system according to claim 7, characterized in that: The method further comprises step five, which specifically comprises: The pose of camera C1 in the camera G coordinate system is recorded as Y1, the pose of camera C2 in the camera G coordinate system is recorded as Y2, and the pose of camera C2 in the camera C1 coordinate system is recorded as Y3. <h2 style=";text-align:left;direction:ltr">Y1<h2 style=";text-align:left;direction:ltr"> -1 <h2 style=";text-align:left;direction:ltr"> Y2-Y3 Among them, the superscript -1 represents the inverse of the matrix; The homogeneous matrix of the pose of the plane target corresponding to camera C1 in the plane target coordinate system corresponding to camera G is recorded as X1, the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera G is recorded as X2, and the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the plane target coordinate system corresponding to camera C1 is recorded as X3, then X1 -1 X2=X3 The homogeneous matrix of the pose of the plane target corresponding to camera G in the camera G coordinate system is recorded as C k' , the homogeneous matrix of the pose of the plane target corresponding to camera C1 in the camera C1 coordinate system is recorded as A k' , the homogeneous matrix of the pose of the plane target corresponding to camera C2 in the camera C2 coordinate system is recorded as B k' ; k'=1,2,…,K', K' is the number of image groups that meet the integrity condition of the three cameras at the same time; Among them, E′ p is an intermediate variable; For the k'th group of images, the reprojection errors of camera G, camera C1 and camera C2 are: in, is the reprojection error of camera G, is the reprojection error of camera C1, is the reprojection error of camera C2, K G is the intrinsic parameter matrix of camera G, P l is the coordinate of the corner point on the plane target corresponding to the camera G in the plane target coordinate system, and They are P l The coordinate calculation values ​​in the image coordinate system of the k'th group of images collected by camera G are reprojected in two different ways, YesP l The actual coordinates corresponding to the image coordinate system of the image captured by camera G, L is the number of corner points corresponding to the plane target detected in the k'th group of images captured by camera G, P l′ is the coordinate of the corner point on the plane target corresponding to camera C1 in the plane target coordinate system, and They are P l′ The coordinate calculation values ​​in the image coordinate system of the k'th group of images acquired by camera C1 are reprojected in two different ways, YesP l′ The actual coordinates corresponding to the image coordinate system of the image captured by camera C1, L′ is the number of corner points corresponding to the planar target detected in the k′th group of images captured by camera C1, P l″ is the coordinate of the corner point on the plane target corresponding to camera C2 in the plane target coordinate system, and They are P l″ The coordinate calculation values ​​in the image coordinate system of the k'th group of images acquired by camera C2 are reprojected in two different ways, YesP l″ The actual coordinates corresponding to the image coordinate system of the image captured by camera C2, L″ is the number of corner points corresponding to the plane target detected in the k'th group of images captured by camera C2; The overall reprojection error is then: The constructed residual function is: E′=λ'E′ P +(1-λ')E′ rj Among them, λ' is the weight coefficient; Take the relative pose result calculated in step 4 as the initial value, use the Levenberg-Marquadt algorithm to minimize the value of the residual function E′, and solve the homogeneous matrices X1, X2, X3 and Y1, Y2, Y3; Taking the camera G coordinate system as the reference coordinate system, the homogeneous matrix of the external parameters of camera G is E 4*4 , E 4*4 is the identity matrix, the homogeneous matrix of the external parameters of camera C1 is Y1, and the homogeneous matrix of the external parameters of camera C2 is Y2.

Images