Multi-camera joint calibration method without common view, medium and device

Abstract

The present application relates to a kind of multi-camera joint calibration method without public view, medium and equipment, by collecting the several two-dimensional images of several ball calibration targets and target tracking instrument conversion matrix;Several three-dimensional point clouds of calibration ball and ball center coordinates are collected;The ball center coordinates of robot end calibration ball and corresponding flange pose under tracking instrument coordinate system are collected;Two-dimensional ball center coordinates in each two-dimensional image are extracted, and three-dimensional ball center coordinates are converted to obtain the external parameter of each 2D camera;According to the coordinates of the ball center of each calibration ball, the external parameter of each 3D camera is solved;Space transformation closed loop equation is constructed;Linear solution of end tool parameter and robot base external parameter is solved, and linear solution is used as initial value to construct the nonlinear least square optimization problem with the goal of minimizing space transformation closed loop equation residual, and the optimal solution of end tool parameter and robot base external parameter is solved, and calibration is completed.Thereby the problem of high-precision collaborative work of multi-source sensor in complex intelligent manufacturing scene is improved.

Description

Technical Field

[0001] This invention relates to the field of equipment calibration technology, and in particular to a method, medium, and device for joint calibration of multiple cameras without a common field of view. Background Technology

[0002] With the advent of "Industry 4.0" and the era of intelligent manufacturing, flexible automation systems centered on machine vision and industrial robots are increasingly being applied in high-end equipment manufacturing, aerospace, and automotive industries. Especially in large-space scenarios such as automotive body-in-white positioning, large composite material component assembly, and aircraft skin inspection, to achieve real-time measurement of complex workpieces across all dimensions and angles, and precise robot operation, it is typically necessary to deploy multiple vision sensors (cameras) to form a distributed vision measurement network. Due to physical space and field-of-view limitations, these cameras are often placed around the workpiece or near key features, resulting in no common field of view between cameras. Simultaneously, to achieve automatic robot guidance based on visual feedback (such as grasping, drilling, and gluing), it is essential to establish a high-precision spatial correlation between the 3D information perceived by the vision measurement network and the motion coordinate system of the robot actuator. This requires unified extrinsic parameter calibration of the entire system—i.e., multiple cameras and robots without a common field of view—to determine the rigid transformation relationship of each coordinate system relative to the common global world coordinate system.

[0003] Existing technologies mainly offer the following types of solutions: The first category is the traditional hand-eye calibration method based on a common calibration object. This method fixes a calibration board with a high-precision known pattern (such as a checkerboard or dot array) to the robot's end effector. By controlling the robot to move the calibration board in multiple poses, a fixed camera observes the calibration board from different angles, and then solves the hand-eye calibration equation of the form "AX=XB" to obtain the transformation matrix of the camera coordinate system relative to the robot's base coordinate system. However, this method requires the calibration board to always be within the camera's field of view, and the size and accuracy of the calibration board directly affect the final result, making it difficult to implement in large-space scenarios.

[0004] The second category is camera calibration methods based on motion structures. For cameras with a common field of view, the principles of motion structure reconstruction (SFM) or binocular stereo vision can be used to establish image correspondences by simultaneously observing multiple feature points in space, thereby calculating the relative pose between cameras. However, this method heavily relies on the overlapping fields of view between cameras and fails completely for camera groups without a common field of view.

[0005] The third category is multi-camera network calibration methods based on ultra-large calibration objects or segmented calibration. For camera networks without a common field of view, one approach is to use an ultra-large calibration object large enough to cover the fields of view of all cameras, allowing all cameras to simultaneously photograph different parts of the calibration object to establish a connection. However, manufacturing and placing large-scale, high-precision ultra-large calibration objects is extremely difficult and costly, and their deformation and measurement errors directly introduce into the system. Another segmented calibration scheme uses movable small calibration boards to calibrate adjacent cameras with a common field of view in pairs, and then connects the entire network through coordinate transfer. However, this "relay" method suffers from serious error accumulation problems; the larger the network, the more severe the degradation of the extrinsic parameters of the end cameras, making it impossible to meet the high-precision requirements of global consistency in large spaces.

[0006] The fourth category is the independent measurement and post-fitting method based on laser trackers. This method uses a laser tracker as the "gold standard" to measure the extrinsic parameters of each camera (e.g., measuring the relationship between the optical center and the target ball by mounting a target ball on the camera) and the position of the robot base, and then fits and aligns them to the global coordinate system in post-processing software. Although the accuracy is high, the process is fragmented, the operation is complex (requiring the design of a special target mounting interface for the camera), and it does not fully utilize the dynamic measurement capabilities of the laser tracker, resulting in insufficient efficiency and systematicity.

[0007] In summary, the existing technology has the following main defects when applied to the calibration of multi-camera-robot systems in large spaces without a common field of view: (1) Contradiction between accuracy and reliability - traditional methods are limited by the field of view and the size of the calibration object, segmented calibration has error accumulation, and the accuracy of the ultra-large calibration object itself is difficult to guarantee; (2) Complex operation and low efficiency - relying on robot motion planning, multiple movement of calibration board or frequent replacement of target, the process is cumbersome and the degree of automation is low; (3) Lack of global consistency - multi-camera calibration and robot hand-eye calibration are usually processed separately, lacking a common high-precision global coordinate system for each sensor, resulting in the visual measurement results not being able to seamlessly connect with the robot motion commands under the same spatial reference; (4) Insufficient flexibility - specific requirements for scene layout, poor adaptability in harsh scenarios where the camera is fixedly installed and the field of view is limited.

[0008] Therefore, there is an urgent need for a unified calibration method that can achieve high precision, high efficiency, and systematicity, so as to lay a reliable spatial reference for subsequent precision vision measurement and autonomous robot operation. Summary of the Invention

[0009] Based on this, the purpose of this application is to provide a multi-camera joint calibration method, medium, and device without a common field of view to solve at least one of the technical problems mentioned in the background art.

[0010] Firstly, this application provides a multi-camera joint calibration method without a common field of view, including: S1: Acquire several two-dimensional images of the multi-sphere calibration target under the field of view of each 2D camera and the target-tracker transformation matrix corresponding to each two-dimensional image; acquire several three-dimensional point clouds of the calibration sphere under the field of view of each 3D camera and the center coordinates of the sphere in the tracker coordinate system corresponding to each three-dimensional point cloud; fix the calibration sphere to the end of the robot and acquire the center coordinates of the calibration sphere of the robot end in the tracker coordinate system and the corresponding flange pose under different poses; S2: Extract the two-dimensional sphere center coordinates from each two-dimensional image, and transform them according to the target-tracker transformation matrix to obtain the three-dimensional sphere center coordinates in the laser tracker coordinate system. Solve for the extrinsic parameters of each 2D camera based on the prior camera intrinsic parameters, the three-dimensional sphere center coordinates, and the two-dimensional sphere center coordinates. S3: Perform geometric fitting on each point cloud data to obtain the coordinates of the center of each calibration sphere in the 3D camera coordinate system. Solve for the extrinsic parameters of each 3D camera based on the coordinates of the center of each calibration sphere in the laser tracker coordinate system and the 3D camera coordinate system. S4: Based on the robot base extrinsic parameters, end-effector parameters, and the coordinates of the center of the robot end-effector calibration ball and the corresponding flange pose in the tracker coordinate system under different poses, construct the spatial transformation closed-loop equation; S5: Based on the relative motion of multiple flanges and the relative motion of the center of the calibration ball of the robot end effector in the coordinate system of the tracker, the linear solutions of the end effector parameters and the robot base extrinsic parameters are obtained. Using the linear solutions as initial values, a nonlinear least squares optimization problem is constructed with the objective of minimizing the residual of the spatial transformation closed-loop equation. The optimal solutions of the end effector parameters and the robot base extrinsic parameters are obtained, and the calibration is completed.

[0011] Furthermore, the steps for obtaining the extrinsic parameters of each 2D camera based on prior camera intrinsic parameters, 3D sphere center coordinates, and 2D sphere center coordinates include: Extract the two-dimensional sphere center coordinates from each two-dimensional image, and transform them into three-dimensional sphere center coordinates in several tracker coordinate systems according to the target-tracker transformation matrix corresponding to each two-dimensional image, thus constructing several sets of 2D-3D feature point pairs; Based on the prior camera intrinsic parameters and 2D-3D feature point pairs, perspective projection geometric constraints are constructed, and the initial extrinsic parameters of each 2D camera are obtained by solving. Based on the initial extrinsic parameters of each 2D camera, a nonlinear least squares optimization model based on minimizing the global reprojection error is constructed, and the optimized extrinsic parameters of each 2D camera are obtained by solving the model.

[0012] Further steps to obtain the optimized extrinsic parameters for each 2D camera include: S231: Based on the current extrinsic parameters of each 2D camera, project the three-dimensional sphere center coordinates onto the corresponding two-dimensional image to obtain several sphere center projection coordinates; S232: Obtain the distance between the projected coordinates of each sphere center and the corresponding 2D sphere center coordinates as the reprojection error, and determine whether the sum of each reprojection error is greater than the error threshold. If not, obtain the optimized extrinsic parameters of each 2D camera. S233: If so, then perform Taylor expansion on the prior pinhole camera model of each 2D camera to obtain the partial derivative between the reprojection error and the current extrinsic parameters of the corresponding camera, and construct the Jacobian matrix; S234: Construct a system of linear equations based on the Jacobian matrix, solve for the extrinsic parameter correction, correct the extrinsic parameters of each 2D camera based on the extrinsic parameter correction, and return to step S231.

[0013] Furthermore, the steps for obtaining the extrinsic parameters of each 3D camera by solving for the coordinates of the center of each calibration sphere in the laser tracker coordinate system and the 3D camera coordinate system include: Construct several 3D feature point pairs based on the coordinates of the center of each calibration sphere in the coordinate system of the laser tracker and the coordinate system of the 3D camera. Construct an objective optimization function that minimizes the sum of distances between corresponding feature points in a 3D feature point pair after they have been transformed to the same coordinate system; Singular value decomposition is performed on the covariance matrix corresponding to the objective optimization function to obtain the rotation matrix and translation vector of each 3D camera, and the extrinsic parameters of each 3D camera are obtained by combining them.

[0014] Furthermore, after obtaining the extrinsic parameters of each 3D camera, the process also includes: Obtain the camera extrinsic parameters corresponding to each 3D point cloud in the same camera coordinate system, and decompose them to obtain the translation vector and rotation matrix of each extrinsic parameter; Each rotation matrix is ​​converted into a quaternion, and the average value of the translation vector and quaternion corresponding to each 3D point cloud is calculated to obtain the average translation vector and average quaternion, which are then converted to obtain several average translation matrices and average rotation matrices. By concatenating the average translation matrix and the corresponding average rotation matrix, the optimized extrinsic parameters of each 3D camera are obtained.

[0015] Furthermore, the steps for obtaining linear solutions to the end-effector parameters and robot base extrinsic parameters include: Based on the collected flange poses, calculate the relative motion matrix between adjacent poses of the flange. Based on the collected coordinates of the ball center in several sets of tracker coordinate systems, calculate the relative motion matrix of the ball center between adjacent poses; Construct a linear equation for hand-eye calibration based on the relative motion matrix of the flange and the relative motion matrix of the sphere's center; Solve the linear equations for hand-eye calibration to obtain linear solutions for the end-effector parameters and the robot base extrinsic parameters.

[0016] Further steps to obtain the optimal solutions for the end-effector parameters and the robot base extrinsic parameters include: Construct a spatial transformation closed-loop equation that includes robot base extrinsic parameters, end effector parameters, flange pose, and sphere center coordinates; Using the linear solution as the initial value, a nonlinear least squares objective function is constructed to minimize the residual norm of the spatial transformation closed-loop equation; The nonlinear least squares objective function is solved iteratively until the iteration termination condition is met, and the optimal solutions for the end-effector parameters and robot base extrinsic parameters are obtained, thus completing the calibration.

[0017] Furthermore, after calibration is completed, it also includes: The laser tracker coordinate system is used as the global world coordinate system; An uncalibrated verification target is placed in the system workspace, and the true coordinates of the verification target are measured using a laser tracker. The target was measured and verified using the calibrated 2D and 3D cameras, and the optimized extrinsic parameters were transformed to the global world coordinate system to obtain the visual measurement coordinates. Calculate the root mean square error between the visual measurement coordinates and the true coordinates, and determine whether it is not greater than the error threshold. If yes, output the calibration result; otherwise, return to step S1.

[0018] Secondly, this application also provides a computer storage medium storing executable program code; the executable program code is used to execute the multi-camera joint calibration method without a common field of view as described in any one of the first aspects.

[0019] Thirdly, this application also provides a terminal device, including a memory and a processor; the memory stores program code executable by the processor; the program code is used to execute the multi-camera joint calibration method without a common field of view as described in any one of the first aspects.

[0020] This application provides a multi-camera joint calibration method, medium, and device without a common field of view. It involves acquiring several two-dimensional images of a multi-sphere calibration target in the fields of view of each 2D camera, along with the target-tracker transformation matrix corresponding to each two-dimensional image; acquiring several three-dimensional point clouds of the calibration sphere in the fields of view of each 3D camera, along with the sphere center coordinates in the tracker coordinate system corresponding to each three-dimensional point cloud; fixing the calibration sphere to the end effector of a robot, and acquiring the sphere center coordinates of the calibration sphere in the tracker coordinate system and the corresponding flange pose under different poses; extracting the two-dimensional sphere center coordinates from each two-dimensional image, and converting them according to the target-tracker transformation matrix to obtain the three-dimensional sphere center coordinates in the laser tracker coordinate system; and solving for the extrinsic parameters of each 2D camera based on the prior camera intrinsic parameters, the three-dimensional sphere center coordinates, and the two-dimensional sphere center coordinates; and then processing each point cloud... Geometric fitting of the data yields the coordinates of the center of each calibration sphere in the 3D camera coordinate system. Based on these coordinates in both the laser tracker and 3D camera coordinate systems, the extrinsic parameters of each 3D camera are obtained. Using the robot base extrinsic parameters, end-effector parameters, and the coordinates of the center of the calibration sphere in the tracker coordinate system under different poses, along with the corresponding flange poses, a spatial transformation closed-loop equation is constructed. Based on the relative motion of multiple flanges and the relative motion of the center of the calibration sphere in the tracker coordinate system, linear solutions for the end-effector parameters and robot base extrinsic parameters are obtained. Using these linear solutions as initial values, a nonlinear least-squares optimization problem is constructed with the objective of minimizing the residuals of the spatial transformation closed-loop equation. Solving this problem yields the optimal solutions for the end-effector parameters and robot base extrinsic parameters, completing the calibration process. This addresses the challenge of high-precision collaborative operation of multiple sensors in complex intelligent manufacturing scenarios. Attached Figure Description

[0021] Figure 1 This is a flowchart of an embodiment of the multi-camera joint calibration method without a common field of view of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., these directional indications are only used to explain the relative positional relationships and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly. Furthermore, if the embodiments of the present invention involve descriptions such as "first," "second," "S1," "S2," "step one," "step two," etc., these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance, or implicitly indicating the number of technical features indicated or the execution order of the method. Those skilled in the art will understand that anything that does not violate the inventive concept should be included within the scope of protection of the present invention.

[0024] like Figure 1 As shown, the present invention provides a multi-camera joint calibration method without a common field of view, comprising: S1: Acquire several two-dimensional images of the multi-sphere calibration target under the field of view of each 2D camera and the target-tracker transformation matrix corresponding to each two-dimensional image; acquire several three-dimensional point clouds of the calibration sphere under the field of view of each 3D camera and the center coordinates of the sphere in the tracker coordinate system corresponding to each three-dimensional point cloud; fix the calibration sphere to the end of the robot and acquire the center coordinates of the calibration sphere of the robot end in the tracker coordinate system and the corresponding flange pose under different poses; Specifically, a laser tracker can be introduced as a high-precision global benchmark to simultaneously acquire heterogeneous data from multiple sources, including 2D images, 3D point clouds, target-tracker transformation matrices, and robot flange poses, thus constructing a complete calibration data system covering 2D cameras, 3D cameras, and the robot's end effector. The multi-ball calibration target design avoids the limitations of traditional calibration boards when the field of view is obstructed, while the design of fixing the calibration balls to the robot's end effector directly links the robot's kinematics and visual coordinate system, providing a solid data foundation for subsequent high-precision calibration across modalities and coordinate systems, ensuring that the calibration process is not limited by the common field of view.

[0025] In a preferred embodiment, the data acquisition steps include: 1. System deployment and integrated data acquisition: The calibration system consists of: multiple cameras (which can be 2D cameras or a 3D camera network) fixed at key locations in a large space with no shared field of view, at least one industrial robot, one high-precision laser tracker, and corresponding calibration targets.

[0026] 1.1: Camera extrinsic parameter correlation data acquisition This step employs different targets and acquisition strategies depending on the camera type, but all are carried out synchronously within the same process framework.

[0027] For 2D cameras: A rigid multi-sphere calibration target is used, on which at least three calibration spheres of known precise diameter and with known relative center positions are fixed. The target is moved so that it sequentially enters the field of view of each 2D camera, and is placed in multiple (N≥3) different positions and poses within each camera's field of view. At each placement pose, the following are performed synchronously: a) The laser tracker measures and records the transformation T_L_T from the target coordinate system {T} to the laser tracker coordinate system {L}; b) The corresponding 2D camera is triggered to capture an image of the target. A dataset {T_L_T, Image_i} is generated for each 2D camera C_2Di.

[0028] For 3D cameras: Use a single calibration sphere (with a known precise diameter). Move the calibration sphere sequentially into the effective measurement range of each 3D camera, placing it at multiple (N≥3) different spatial locations within the field of view of each camera. At each location, simultaneously perform: a) The laser tracker directly measures and records the coordinates P_L of the calibration sphere's center in the {L} system; b) Trigger the corresponding 3D camera to acquire point cloud data containing the sphere. Generate a dataset {P_L, PointCloud_j} for each 3D camera C_3Dj.

[0029] 1.2: Robot External Parameter Correlation Data Acquisition This step is independent of camera type and is a unified operation of the system. A calibration sphere is securely mounted on the robot's end effector flange, forming the end-effector measurement tool. The robot is controlled to move the end effector tool to multiple (M≥10) different poses. At each robot pose, the following are executed synchronously: a) The laser tracker measures the coordinates Q_L of the end effector calibration sphere's center in the {L} coordinate system; b) The robot controller records the pose T_R_Flange of the end effector flange in the base coordinate system {R}. A robot dataset {Q_L, T_R_Flange} is generated.

[0030] S2: Extract the two-dimensional sphere center coordinates from each two-dimensional image, and transform them according to the target-tracker transformation matrix to obtain the three-dimensional sphere center coordinates in the laser tracker coordinate system. Solve for the extrinsic parameters of each 2D camera based on the prior camera intrinsic parameters, the three-dimensional sphere center coordinates, and the two-dimensional sphere center coordinates. Specifically, the high-precision 3D measurement capabilities of a laser tracker can be utilized to convert the sphere center coordinates in the 2D image into 3D coordinates in a global coordinate system, constructing 2D-3D feature point pairs. Then, combined with prior camera intrinsic parameters, the initial extrinsic parameters are solved using perspective projection geometric constraints. Finally, the extrinsic parameters are optimized by minimizing the global reprojection error. This method avoids the limitations of traditional 2D camera calibration, which relies on a common field of view or manual marking. Leveraging the global reference characteristics of the laser tracker, it achieves high-precision solution of 2D camera extrinsic parameters without a common field of view. Furthermore, nonlinear optimization further reduces reprojection error and improves the robustness of the extrinsic parameters.

[0031] In a preferred embodiment, the step of obtaining the extrinsic parameters of each 2D camera based on prior camera intrinsic parameters, three-dimensional sphere center coordinates, and two-dimensional sphere center coordinates includes: S21: Extract the coordinates of the two-dimensional sphere center in each two-dimensional image, and according to the target-tracker transformation matrix corresponding to each two-dimensional image, transform to obtain the coordinates of the three-dimensional sphere center in several tracker coordinate systems, and construct several sets of 2D-3D feature point pairs; S22: Construct perspective projection geometric constraints based on prior camera intrinsic parameters and 2D-3D feature point pairs, and solve for the initial extrinsic parameters of each 2D camera; S23: Construct a nonlinear least squares optimization model based on minimizing global reprojection error according to the initial extrinsic parameters of each 2D camera, and solve for the optimized extrinsic parameters of each 2D camera.

[0032] Specifically, the high-precision measurement capabilities of laser trackers can be utilized to directly map feature points in 2D images to the global 3D space, constructing a cross-modal 2D-3D feature association. This "dimensionality-upgrading" approach cleverly bypasses the limitation of traditional multi-camera calibration relying on a "common field of view," effectively improving the flexibility of the calibration layout. Then, perspective projection geometric constraints (optionally a PnP problem) are constructed based on prior camera intrinsic parameters and 2D-3D feature point pairs. The initial pose of the camera is quickly calculated using known intrinsic parameters and rigid body transformation relationships, providing an initial solution close to the true value for subsequent nonlinear optimization. This effectively avoids the risk of iterative algorithms diverging or getting trapped in local optima due to excessive initial value deviations, ensuring the convergence speed and stability of the calibration process. Finally, based on the obtained initial solution, a global reprojection error minimization strategy is introduced. All observation data are jointly optimized using the nonlinear least squares method, improving the local accuracy of a single camera to the global optimal accuracy of the entire system, significantly improving the robustness and accuracy of the 2D camera extrinsic parameters.

[0033] In a preferred embodiment, the step of solving for the optimized extrinsic parameters of each 2D camera includes: S231: Based on the current extrinsic parameters of each 2D camera, project the three-dimensional sphere center coordinates onto the corresponding two-dimensional image to obtain several sphere center projection coordinates; S232: Obtain the distance between the projected coordinates of each sphere center and the corresponding 2D sphere center coordinates as the reprojection error, and determine whether the sum of each reprojection error is greater than the error threshold. If not, obtain the optimized extrinsic parameters of each 2D camera. S233: If so, then perform Taylor expansion on the prior pinhole camera model of each 2D camera to obtain the partial derivative between the reprojection error and the current extrinsic parameters of the corresponding camera, and construct the Jacobian matrix; S234: Construct a system of linear equations based on the Jacobian matrix, solve for the extrinsic parameter correction, correct the extrinsic parameters of each 2D camera based on the extrinsic parameter correction, and return to step S231.

[0034] Specifically, the position of feature points on the image can be predicted using a forward projection model, providing a benchmark for error calculation. This allows the algorithm to quantify the deviation between the "theoretical position" and the "actual observation." Then, by setting an error threshold, a convergence criterion is constructed to automatically terminate the calibration process, avoiding unnecessary excessive iterations and saving computing resources. Simultaneously, it ensures that the final result meets the preset accuracy requirements, guaranteeing a balance between calibration efficiency and accuracy. Furthermore, Taylor expansion is used to locally linearize the complex nonlinear reprojection error problem. By constructing a Jacobian matrix, the gradient of the influence of external parameter changes on the error is accurately described. The correction amount is solved using a system of linear equations and iteratively updated, efficiently correcting the external parameters to the optimal value. In optimal conditions, this method addresses the challenge of solving nonlinear least squares problems. It accurately extracts the coordinates of the calibration sphere's center by geometric fitting of the 3D point cloud. Then, it constructs 3D feature point pairs using the center coordinates in the laser tracker's coordinate system and the 3D camera's coordinate system. The extrinsic parameters are solved by minimizing the distance between these feature points using an objective optimization function. Finally, quaternion averaging further improves the accuracy of the extrinsic parameters. This approach eliminates the need for a shared field of view between the 3D camera and the 2D camera or the robot's end effector; it relies solely on the global coordinate association of the laser tracker to achieve accurate calibration of the 3D camera's extrinsic parameters. Furthermore, the combination of geometric fitting and quaternion averaging effectively suppresses the interference of point cloud noise on the extraction of the sphere's center coordinates, ensuring the accuracy of the extrinsic parameter solution.

[0035] In a preferred embodiment, the step of obtaining the extrinsic parameters of each 3D camera based on the coordinates of the center of each calibration sphere in the laser tracker coordinate system and the 3D camera coordinate system includes: S31: Construct several pairs of 3D feature points based on the coordinates of the center of each calibration sphere in the coordinate system of the laser tracker and the coordinate system of the 3D camera; S32: Construct an objective optimization function with the goal of minimizing the sum of distances between corresponding feature points after transforming each feature point in a 3D feature point pair to the same coordinate system; S33: Perform singular value decomposition on the covariance matrix corresponding to the objective optimization function to obtain the rotation matrix and translation vector of each 3D camera, and combine them to obtain the extrinsic parameters of each 3D camera.

[0036] Specifically, a direct spatial mapping relationship can be established between the global coordinate system (tracker) and the local coordinate system (3D camera). By matching the expression of the center of the same calibration sphere in different coordinate systems, the most direct geometric constraint is provided for solving the rigid body transformation, eliminating the cumulative error that may be introduced by the intermediate transformation. Then, the objective function is to use "minimum sum of distances" to find the optimal rigid body transformation to align the two sets of point clouds. Since the objective of point-to-distance optimization has a clear physical meaning and is robust to noise, it can effectively handle data fluctuations caused by the noise of 3D camera depth measurement and ensure the stability of the extrinsic parameter solution. Finally, singular value decomposition (SVD) is used to directly solve the rotation and translation matrices, thereby quickly and accurately extracting the orthogonal rotation and translation components from the covariance matrix, avoiding the problem of non-orthogonalization of the rotation matrix and ensuring the mathematical rigor of the extrinsic parameters.

[0037] In a preferred embodiment, after obtaining the extrinsic parameters of each 3D camera, the process includes: S331: Obtain the camera extrinsic parameters corresponding to each 3D point cloud in the same camera coordinate system, and decompose them to obtain the translation vector and rotation matrix of each extrinsic parameter; S332: Convert each rotation matrix into a quaternion, and calculate the average value of the translation vector and quaternion corresponding to each 3D point cloud to obtain the average translation vector and average quaternion, and convert them to obtain several average translation matrices and average rotation matrices. S333: Concatenate the average translation matrices and the corresponding average rotation matrices to obtain the optimized extrinsic parameters for each 3D camera.

[0038] Specifically, since 3D point cloud data is easily affected by factors such as speckle noise and ambient light, the extrinsic parameters of a single measurement may fluctuate. Therefore, we can choose to convert the rotation matrix into quaternions for averaging, and calculate the average value of the translation vector and quaternion corresponding to each 3D point cloud to obtain the average translation vector and average quaternion. This results in several average translation matrices and average rotation matrices, which can avoid the gimbal lock problem of Euler angles and can handle the averaging of rotation data more smoothly. Through averaging in the time domain or spatial domain, random noise is effectively suppressed, and the signal-to-noise ratio of the extrinsic parameter estimation is improved. Finally, we concatenate the average translation matrices and the corresponding average rotation matrices to obtain the optimized extrinsic parameters of each 3D camera. By recombining the optimized rotation and translation components into a complete rigid body transformation matrix, the extrinsic parameters are refined. By fusing information from multiple observations, the reliability of the calibration is further consolidated, making the final extrinsic parameters more representative of the camera's stable state in the real environment, rather than the accidental result of a specific measurement, thus improving data robustness.

[0039] In another preferred embodiment, the steps for solving for each 2D camera and 3D camera include: 2. Camera network extrinsic parameter calculation. Based on the data collected in step 1, the extrinsic parameters of each camera in the camera network are calculated independently.

[0040] 2.1: 2D Camera Extrinsic Parameter Calculation For each 2D camera C_2Di and its dataset {T_L_T, Image_i}: Image feature extraction and spatial point mapping: Extract the image point coordinates of each sphere on the multi-sphere target from each image; using the synchronously recorded T_L_T and the known coordinates of the calibrated sphere centers on the target, calculate the three-dimensional coordinates P_L of each sphere center in the {L} system corresponding to each image.

[0041] Single-frame pose estimation: By using the camera intrinsic parameters and the corresponding image point coordinates and the relationship between P_L, the initial estimate of the camera extrinsic parameters T_L_C2Di for each image is obtained by solving the perspective n-point (PnP) problem.

[0042] Global Bundle Adjustment (BA) optimization: All pose observation data are input into a bundle adjustment (BA) optimization framework. With the goal of minimizing image reprojection error, the high-precision, globally consistent extrinsic parameters T_L_C2Di of the 2D camera are obtained through joint optimization.

[0043] 2.2: 3D Camera Extrinsic Parameter Calculation For each 3D camera C_3Dj and its dataset {P_L, PointCloud_j}: Point cloud sphere center fitting: Segment and geometric fitting are performed on each frame of point cloud to extract the three-dimensional coordinates P_C3Dj of the calibrated sphere center in the 3D camera's own coordinate system {C_3Dj}.

[0044] Absolute orientation solution: For each corresponding 3D point pair (P_L, P_C3Dj), the absolute orientation problem is solved by using methods such as singular value decomposition (SVD) to calculate the optimal rotation matrix R and translation vector t, such that P_L = R * P_C3Dj + t. This yields the extrinsic transformation matrix T_L_C3Dj for the 3D camera.

[0045] Multi-location fusion: By combining the results from multiple locations, the robustness and accuracy of the extrinsic parameter T_L_C3Dj are further improved through averaging or nonlinear optimization.

[0046] S4: Based on the robot base extrinsic parameters, end-effector parameters, and the coordinates of the center of the robot end-effector calibration ball and the corresponding flange pose in the tracker coordinate system under different poses, construct the spatial transformation closed-loop equation; S5: Based on the relative motion of multiple flanges and the relative motion of the center of the calibration ball of the robot end effector in the coordinate system of the tracker, the linear solutions of the end effector parameters and the robot base extrinsic parameters are obtained. Using the linear solutions as initial values, a nonlinear least squares optimization problem is constructed with the objective of minimizing the residual of the spatial transformation closed-loop equation. The optimal solutions of the end effector parameters and the robot base extrinsic parameters are obtained, and the calibration is completed.

[0047] Specifically, for each different pose, the coordinates of the center of the robot's end-effector calibration ball in the tracker coordinate system and the corresponding flange pose can be selected. Based on the robot base extrinsic parameters and end-effector parameters, a spatial transformation closed-loop equation is constructed. Then, by constructing linear equations of the relative motion of the flange and the relative motion of the ball center, the initial solutions of the end-effector parameters and the robot base extrinsic parameters are quickly obtained. Then, using the linear solutions as initial values, a nonlinear least squares optimization problem is constructed with the objective of minimizing the residual of the spatial transformation closed-loop equation. The optimal solutions of the end-effector parameters and the robot base extrinsic parameters are obtained, and the calibration is completed. Through the two-stage solution strategy of "linear initial value + nonlinear optimization", the problem of nonlinear optimization getting trapped in local optima due to poor initial values ​​is avoided. Furthermore, the accuracy of the end-effector parameters and the robot base extrinsic parameters is further improved by minimizing the residual of the closed-loop equation.

[0048] In a preferred embodiment, the step of solving for linear solutions to the end-effector parameters and the robot base extrinsic parameters includes: S51: Calculate the relative motion matrix of the flanges between adjacent poses based on the collected flange poses. S52: Calculate the relative motion matrix of the ball centers between adjacent poses based on the collected coordinates of the ball centers in several sets of tracker coordinate systems; S53: Construct a linear equation for hand-eye calibration based on the relative motion matrix of the flange and the relative motion matrix of the sphere center; S54: Solve the linear equations for hand-eye calibration to obtain linear solutions for the end-effector parameters and the robot base extrinsic parameters.

[0049] Specifically, relative displacement information can be extracted from robot kinematic data. By using relative motion instead of absolute coordinates, the influence of the robot base coordinate system definition deviation on subsequent calculations can be effectively eliminated, transforming the problem into a pure rigid body motion correlation problem. This simplifies the complexity of the mathematical model. Simultaneously, high-precision relative motion data measured by a laser tracker can be extracted. Using the laser tracker as a "true value" reference, its measured relative motion reflects the true trajectory of the robot end effector in physical space, providing high-precision observation data for hand-eye calibration. Then, a hand-eye calibration model can be constructed based on any existing method, linking robot motion with external measurement data. Initial solutions can be obtained by solving linear equations, which can quickly estimate the end effector parameters and the extrinsic parameters of the robot base relative to the world coordinate system. This provides initial values ​​for subsequent high-precision nonlinear optimization and prevents the optimization process from getting trapped in local minima.

[0050] In a preferred embodiment, the step of obtaining the optimal solutions for the end-effector parameters and the robot base extrinsic parameters includes: S55: Construct a spatial transformation closed-loop equation that includes robot base extrinsic parameters, end effector parameters, flange pose, and sphere center coordinates; S56: Using the linear solution as the initial value, construct a nonlinear least squares objective function that minimizes the residual norm of the spatial transformation closed-loop equation; S57: Iterate through the nonlinear least squares objective function until the iteration termination condition is met, obtain the optimal solutions for the end-effector parameters and the robot base extrinsic parameters, and complete the calibration.

[0051] Specifically, a complete spatial transformation closed loop can be established, from the robot base to the end effector, then to the calibration ball, and finally back to the tracker coordinate system. This comprehensively describes the geometric constraints of each component in the system, ensuring the self-consistency of the calibration results in the topological structure. This is the core mathematical model for improving the overall system accuracy. Then, nonlinear iterative optimization is performed based on the initial values ​​until the iteration termination condition is met, obtaining the optimal solutions for the end effector parameters and the robot base extrinsic parameters, thus completing the calibration. This minimizes the residuals of the closed-loop equations (i.e., the difference between the theoretical position and the actual measured position), thereby simultaneously compensating for robot kinematic errors, installation errors, and measurement noise. Through a global optimization strategy, the accuracy of the end effector parameters and the base extrinsic parameters is improved to the system limit, achieving perfect coordination between the robot's "hand" (end effector) and "eye" (vision / tracker system).

[0052] In another preferred embodiment, the optimal solutions for the end-effector parameters and robot base extrinsic parameters are obtained, and the calibration process includes: 3. Joint calculation of robot base extrinsic parameters and end-effector parameters. Using the robot dataset {Q_L, T_R_Flange} collected in step 1, the robot base extrinsic parameters and end effector parameters are solved simultaneously.

[0053] Hand-eye calibration model establishment: For the k-th robot pose, establish the spatial transformation closed-loop equation: T_L_R * T_R_Flange_k * T_Flange_Ball = Q_L_k. Where T_L_R (base extrinsic parameters) and T_Flange_Ball (tool parameters) are constant unknowns to be determined, and T_R_Flange_k and Q_L_k are known observations. This equation is a standard hand-eye calibration model in the form "AX=ZB".

[0054] The two-step solution is as follows: First, by calculating the relative motion of multiple robot flanges and the relative motion of the sphere centers measured by the laser tracker, linear solutions for T_Flange_Ball and T_L_R are obtained. Then, using these solutions as initial values, a nonlinear least-squares optimization problem is constructed to minimize the residuals of the closed-loop equations under all poses, ultimately obtaining high-precision optimal estimates of T_L_R and T_Flange_Ball.

[0055] In a preferred embodiment, after calibration is completed, the method further includes: S61: The laser tracker coordinate system is used as the global world coordinate system; S62: Place an uncalibrated verification target in the system workspace and use a laser tracker to measure the true coordinates of the verification target; S63: Measure and verify the target using the calibrated 2D and 3D cameras, and transform the target to the global world coordinate system based on the optimized extrinsic parameters to obtain the visual measurement coordinates; S64: Calculate the root mean square error between the visual measurement coordinates and the true coordinates, and determine whether it is not greater than the error threshold. If yes, output the calibration result; otherwise, return to step S1.

[0056] Specifically, an optional verification target not involved in the calibration can be introduced. Using the true coordinates measured by the laser tracker as a benchmark, the root mean square error between the visually measured coordinates and the true coordinates is compared to achieve a quantitative assessment of the calibration accuracy. This method avoids the overfitting problem of traditional calibration accuracy assessment relying on the data involved in the calibration. By independently verifying the target, it truly reflects the measurement accuracy of the system in the actual working scenario, providing an objective evaluation standard for the reliability of the calibration results. It then determines whether the error is not greater than the threshold. If so, the calibration result is output; otherwise, it returns to step S1. This iterative process based on the evaluation results further improves the overall accuracy of the system.

[0057] In a preferred embodiment, the verification step after calibration further includes step 4: global coordinate system establishment and system accuracy verification. 4.1 Establishing the Global Coordinate System: The laser tracker coordinate system {L} is defined as the global world coordinate system {World} for the entire multi-camera-robot system. At this point, all sensors are associated with this global coordinate system through precise transformation relationships: Each 2D camera: T_L_C2Di 3D cameras: T_L_C3Dj Robot Base: T_L_R Robotics tool: T_Flange_Ball 4.2 System closed-loop verification: Place the verification ball or target at a new, uncalibrated location within the system workspace. Measure its coordinates using a laser tracker as the baseline true value P_true. Schedule relevant cameras to perform measurements, and use the calibrated extrinsic parameters to transform the measurement results to the {L} coordinate system, obtaining the visual measurement value P_vision. Calculate the deviation between P_vision and P_true (e.g., root mean square error RMSE) to quantitatively evaluate the overall calibration accuracy and reliability of the entire multi-sensor system in a unified global coordinate system.

[0058] In this embodiment, a multi-camera joint calibration method without a common field of view is presented, the core inventive point of which is: 1. A laser tracker is introduced as a global reference. The extrinsic parameters of the 2D camera are solved through 2D-3D projection constraints and the extrinsic parameters of the 3D camera are solved through 3D-3D geometric registration. The parameters of the base and end effector are solved by using robot kinematics correlation. Through the calibration architecture of "global reference + multimodal decoupling", the limitation of common field of view is broken and flexible layout is achieved: Traditional multi-camera calibration usually requires all cameras to see the same calibration board at the same time (i.e., there is a common field of view), which is extremely difficult to achieve in large vehicle body inspection or complex workstations. Therefore, the high-precision ranging capability of the laser tracker can be used to use the spatial coordinates of the calibration ball as an "intermediate bridge", so that the 2D camera and the 3D camera do not need to see each other, but only need to observe the calibration ball. Through the non-contact and non-common-view characteristics, the spatial constraints of sensor layout are broken, which is suitable for complex industrial sites with large field of view and multiple angles.

[0059] 2. Multimodal data fusion: For 2D cameras, perspective projection constraints are constructed using "2D sphere center + 3D coordinates" to fully utilize image texture features; for 3D cameras, rigid body transformation constraints are constructed using "point cloud sphere center + 3D coordinates" to fully utilize spatial geometric features. Through the strategy of solving each mode independently, the optimal accuracy of each sensor is ensured, and high-precision unification of the entire system is achieved.

[0060] 3. Unified parameters across the entire chain: By introducing the joint solution of robot kinematic parameters (base extrinsic parameters and end-effector parameters), the visual coordinate system and the robot control coordinate system are tightly coupled, eliminating the cumulative error caused by too many coordinate system transformation levels in traditional step-by-step calibration, and providing a solid geometric benchmark for subsequent robot guidance operations.

[0061] 4. A two-stage solution strategy of "linear initial value estimation + nonlinear iterative optimization" is adopted. For the 2D camera, the Jacobian matrix is ​​constructed using Taylor expansion, and for the robot parameters, a spatial transformation closed-loop equation is constructed. Through the hierarchical and progressive optimization logic, compared with a single linear solution or direct nonlinear fitting, it has unique technical effects such as fast convergence speed, strong resistance to local optima, and high numerical stability. Specifically, it avoids local optima and ensures global convergence: Nonlinear least squares problems are very sensitive to initial values. Excessive deviation of initial values ​​can easily lead to algorithm divergence or getting trapped in local minima. In this embodiment, an initial solution close to the true value is quickly calculated through linear equations (such as SVD decomposition and hand-eye calibration linear equations). Iteration is then carried out from this starting point. Through the "coarse-to-fine" strategy, it is ensured that the optimization process always takes place in the convex region near the true value, which greatly improves the robustness and success rate of the algorithm.

[0062] 4. High-precision approximation: After obtaining the initial value, a nonlinear objective function based on the reprojection error or closed-loop residual is constructed. The Gauss-Newton method or LM algorithm is used for iterative correction. The Jacobian matrix is ​​calculated by Taylor expansion, which accurately describes the gradient effect of external parameter changes on the projection error, so that each iteration can approximate along the direction of the fastest error reduction.

[0063] 5. Enhanced closed-loop constraints improve system consistency: In solving robot parameters, a spatial transformation closed-loop equation is constructed that includes the base, end effector, tool, and observation data. This forces the rigid body motion constraints of the physical world to be satisfied. By minimizing the closed-loop residual, measurement noise and kinematic errors can be effectively adjusted, ensuring a perfect match between the robot's motion trajectory and the visual observation results.

[0064] 6. A multi-frame data averaging optimization and independent verification target evaluation mechanism is introduced. The rotation component is processed by quaternion averaging, and the system accuracy is quantified by the root mean square error. Through the quality assurance system of "statistical denoising + independent verification", compared with the method of single measurement calibration or self-evaluation relying on calibration residuals, it has unique technical effects with strong noise resistance, objective evaluation and good traceability. Specifically, it is reflected in: suppressing random noise and improving the stability of external parameters: 3D point cloud data is susceptible to speckle noise and ambient light interference. The center coordinates of a single acquisition may have slight jitter. This embodiment avoids the Euler angle gimbal lock problem by converting the rotation matrix into quaternions for averaging, and can more smoothly integrate multi-frame data. Through statistical averaging in the time domain, random noise is effectively filtered out, so that the final external parameters can better represent the stable state of the system, rather than the result of a random measurement, which significantly improves the repeatability accuracy of calibration.

[0065] 7. Independent Truth Value Verification: Traditional calibration often only considers the internal indicator of "reprojection error", which can easily lead to overfitting (i.e., the calibration data error is small, but the actual measurement error is large). By introducing a verification target that was not involved in the calibration, the true coordinates measured by the laser tracker are used as the "gold standard" to calculate the root mean square error of the visual measurement value, thereby objectively and truthfully reflecting the system's measurement capability in the actual workspace.

[0066] This solution, through "dynamic synchronization" and "heterogeneous fusion," goes beyond the relative relationships within the robot and static camera network stitching. Instead, it uses a laser tracker to map the moving robot and the fixed camera into an absolute spatial coordinate system under the premise of time synchronization, thus solving the problem of high-precision collaborative operation of multiple source sensors in complex intelligent manufacturing scenarios.

[0067] On the other hand, the present invention also provides a computer storage medium storing executable program code; the executable program code is used to execute any of the above-mentioned multi-camera joint calibration methods without a common field of view.

[0068] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned multi-camera joint calibration methods without a common field of view.

[0069] For example, the program code can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the program code in the terminal device.

[0070] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the terminal device may also include input / output devices, network access devices, buses, etc.

[0071] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0072] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. The memory can also be an external storage device of the terminal device, such as a plug-in hard drive, SmartMediaCard (SMC), Secure Digital (SD) card, or FlashCard. Furthermore, the memory can include both internal and external storage units of the terminal device. The memory is used to store the program code and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output.

[0073] The aforementioned computer storage medium and terminal device are created based on the aforementioned multi-camera joint calibration method without a common field of view. Their technical functions and beneficial effects will not be elaborated here. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0074] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A multi-camera joint calibration method without a common field of view, characterized in that, include: S1: Acquire several two-dimensional images of the multi-sphere calibration target under the field of view of each 2D camera and the target-tracker transformation matrix corresponding to each two-dimensional image; acquire several three-dimensional point clouds of the calibration sphere under the field of view of each 3D camera and the center coordinates of the sphere in the tracker coordinate system corresponding to each three-dimensional point cloud; fix the calibration sphere to the end of the robot and acquire the center coordinates of the calibration sphere of the robot end in the tracker coordinate system and the corresponding flange pose under different poses; S2: Extract the two-dimensional sphere center coordinates from each two-dimensional image, and transform them according to the target-tracker transformation matrix to obtain the three-dimensional sphere center coordinates in the laser tracker coordinate system. Solve for the extrinsic parameters of each 2D camera based on the prior camera intrinsic parameters, the three-dimensional sphere center coordinates, and the two-dimensional sphere center coordinates. S3: Perform geometric fitting on each point cloud data to obtain the coordinates of the center of each calibration sphere in the 3D camera coordinate system. Solve for the extrinsic parameters of each 3D camera based on the coordinates of the center of each calibration sphere in the laser tracker coordinate system and the 3D camera coordinate system. S4: Based on the robot base extrinsic parameters, end-effector parameters, and the coordinates of the center of the robot end-effector calibration ball and the corresponding flange pose in the tracker coordinate system under different poses, construct the spatial transformation closed-loop equation; S5: Based on the relative motion of multiple flanges and the relative motion of the center of the calibration ball of the robot end effector in the coordinate system of the tracker, the linear solutions of the end effector parameters and the robot base extrinsic parameters are obtained. Using the linear solutions as initial values, a nonlinear least squares optimization problem is constructed with the objective of minimizing the residual of the spatial transformation closed-loop equation. The optimal solutions of the end effector parameters and the robot base extrinsic parameters are obtained, and the calibration is completed.

2. The method according to claim 1, characterized in that, The steps for obtaining the extrinsic parameters of each 2D camera based on prior camera intrinsic parameters, 3D sphere center coordinates, and 2D sphere center coordinates include: Extract the two-dimensional sphere center coordinates from each two-dimensional image, and transform them into three-dimensional sphere center coordinates in several tracker coordinate systems according to the target-tracker transformation matrix corresponding to each two-dimensional image, thus constructing several sets of 2D-3D feature point pairs; Based on the prior camera intrinsic parameters and 2D-3D feature point pairs, perspective projection geometric constraints are constructed, and the initial extrinsic parameters of each 2D camera are obtained by solving. Based on the initial extrinsic parameters of each 2D camera, a nonlinear least squares optimization model based on minimizing the global reprojection error is constructed, and the optimized extrinsic parameters of each 2D camera are obtained by solving the model.

3. The method according to claim 2, characterized in that, The steps to obtain the optimized extrinsic parameters for each 2D camera include: S231: Based on the current extrinsic parameters of each 2D camera, project the three-dimensional sphere center coordinates onto the corresponding two-dimensional image to obtain several sphere center projection coordinates; S232: Obtain the distance between the projected coordinates of each sphere center and the corresponding 2D sphere center coordinates as the reprojection error, and determine whether the sum of each reprojection error is greater than the error threshold. If not, obtain the optimized extrinsic parameters of each 2D camera. S233: If so, then perform Taylor expansion on the prior pinhole camera model of each 2D camera to obtain the partial derivative between the reprojection error and the current extrinsic parameters of the corresponding camera, and construct the Jacobian matrix; S234: Construct a system of linear equations based on the Jacobian matrix, solve for the extrinsic parameter correction, correct the extrinsic parameters of each 2D camera based on the extrinsic parameter correction, and return to step S231.

4. The method according to claim 1, characterized in that, The steps for obtaining the extrinsic parameters of each 3D camera based on the coordinates of the center of each calibration sphere in the laser tracker coordinate system and the 3D camera coordinate system include: Construct several 3D feature point pairs based on the coordinates of the center of each calibration sphere in the coordinate system of the laser tracker and the coordinate system of the 3D camera. Construct an objective optimization function that minimizes the sum of distances between corresponding feature points in a 3D feature point pair after they have been transformed to the same coordinate system; Singular value decomposition is performed on the covariance matrix corresponding to the objective optimization function to obtain the rotation matrix and translation vector of each 3D camera, and the extrinsic parameters of each 3D camera are obtained by combining them.

5. The method according to claim 4, characterized in that, After obtaining the extrinsic parameters of each 3D camera, the process also includes: Obtain the camera extrinsic parameters corresponding to each 3D point cloud in the same camera coordinate system, and decompose them to obtain the translation vector and rotation matrix of each extrinsic parameter; Each rotation matrix is ​​converted into a quaternion, and the average value of the translation vector and quaternion corresponding to each 3D point cloud is calculated to obtain the average translation vector and average quaternion, which are then converted to obtain several average translation matrices and average rotation matrices. By concatenating the average translation matrix and the corresponding average rotation matrix, the optimized extrinsic parameters of each 3D camera are obtained.

6. The method according to claim 1, characterized in that, The steps for obtaining linear solutions to the end-effector parameters and robot base extrinsic parameters include: Based on the collected flange poses, calculate the relative motion matrix between adjacent poses of the flange. Based on the collected coordinates of the ball center in several sets of tracker coordinate systems, calculate the relative motion matrix of the ball center between adjacent poses; Construct a linear equation for hand-eye calibration based on the relative motion matrix of the flange and the relative motion matrix of the sphere's center; Solve the linear equations for hand-eye calibration to obtain linear solutions for the end-effector parameters and the robot base extrinsic parameters.

7. The method according to claim 6, characterized in that, The steps to obtain the optimal solutions for the end-effector parameters and the robot base extrinsic parameters include: Construct a spatial transformation closed-loop equation that includes robot base extrinsic parameters, end effector parameters, flange pose, and sphere center coordinates; Using the linear solution as the initial value, a nonlinear least squares objective function is constructed to minimize the residual norm of the spatial transformation closed-loop equation; The nonlinear least squares objective function is solved iteratively until the iteration termination condition is met, and the optimal solutions for the end-effector parameters and robot base extrinsic parameters are obtained, thus completing the calibration.

8. The method according to any one of claims 1-7, characterized in that, After calibration is completed, it also includes: The laser tracker coordinate system is used as the global world coordinate system; An uncalibrated verification target is placed in the system workspace, and the true coordinates of the verification target are measured using a laser tracker. The target was measured and verified using the calibrated 2D and 3D cameras, and the optimized extrinsic parameters were transformed to the global world coordinate system to obtain the visual measurement coordinates. Calculate the root mean square error between the visual measurement coordinates and the true coordinates, and determine whether it is not greater than the error threshold. If yes, output the calibration result; otherwise, return to step S1.

9. A computer storage medium, characterized in that, It stores executable program code; the executable program code is used to execute the multi-camera joint calibration method without a common field of view as described in any one of claims 1 to 8.

10. A terminal device, characterized in that, It includes a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute the multi-camera joint calibration method without a common field of view as described in any one of claims 1 to 8.

Images