A multi-view joint calibration method for large and complex workpieces

Abstract

The application discloses a multi-view joint calibration method for large and complex workpieces. First, multiple CMOS cameras are selected to form a multi-view vision unit with a common field of view; then, the internal parameter matrix of each camera and the external parameter matrix between adjacent converging cameras are calculated through a checkerboard calibration; afterwards, target workpiece images are collected, and the three-dimensional coordinates of workpiece surface feature points are calculated through the internal parameter matrix; then, linear equations are constructed according to the equal distance property of feature points in the world coordinate system and the camera coordinate system, and the equations are solved by using the least square method; finally, the matrix representation of the feature points in the world coordinate system and the camera coordinate system is established by combining singular value decomposition to remove the centroid, and the pose relationship matrix of all auxiliary cameras relative to the main camera is reversely calculated. The application fully considers the actual problems of insufficient field of view and regional occlusion of large and complex workpieces in the imaging process, effectively improves the field of view range through the construction of a multi-view vision unit, and has strong practical value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision joint calibration, and relates to a multi-view joint calibration method, in particular to a multi-view joint calibration method for large and complex workpieces. BACKGROUND

[0002] Camera calibration as the basis of visual measurement and three-dimensional reconstruction provides a prerequisite for the application of computer vision in many fields, and high-precision measurement and positioning of target workpieces by using camera calibration technology has become a new research direction. A stereo vision system composed of two CMOS cameras has the advantages of simple structure, accurate calibration, and rapid implementation of calibration measurement, and is widely used in industrial detection, workpiece positioning and robot guidance. However, in the calibration of large and complex workpieces, the size of the workpiece far exceeds the calibration range of the binocular stereo vision system, and the entire features of the target workpiece cannot be obtained at the same time. The computer vision joint calibration technology proposed in the application breaks through the field of view limitation of traditional binocular stereo vision, structured light scanner and laser range finder, and improves the calibration accuracy and algorithm reliability. SUMMARY

[0003] In order to solve the actual problems of insufficient field of view and regional occlusion in the process of calibrating large and complex workpieces by using traditional stereo vision systems, the application proposes a multi-view calibration method for large and complex workpieces. The method assembles a multi-view vision unit based on a converging binocular camera; by calibrating the intrinsic parameter matrix, rotation matrix and translation vector of the camera, the pose relationship of the auxiliary camera relative to the main camera is inversely calculated, and finally the joint calibration of all cameras under multi-view in the world coordinate system of the main camera is realized.

[0004] In order to achieve the above purpose, the technical scheme adopted by the application is:

[0005] A multi-view joint calibration method for large and complex workpieces, characterized by comprising the following steps:

[0006] Step 1, selecting N CMOS cameras to form a multi-view vision unit, adjusting the different viewing angles of the cameras to ensure that the field of view covers the entire target workpiece, N being greater than or equal to 3, and assembling the multi-view vision unit in a converging manner with every two adjacent cameras, the multi-view vision unit having a common field of view covering the surface of the target workpiece;

[0007] Step 2, defining the first camera as the main camera C1, and the remaining cameras are arranged in sequence according to the serial number, all being auxiliary cameras C i, i is valued between 2 to N, wherein the auxiliary camera C2 is convergently installed with the main camera C1, and every two adjacent auxiliary cameras are also convergently installed; a sequence of checkerboard images is collected under a common field of view, and the camera parameter matrix of the multi-view vision unit is calculated by calibrating the checkerboard, wherein the camera parameter matrix comprises an internal parameter matrix of each camera and an external parameter matrix between every two adjacent cameras;

[0008] Step 3, according to the camera parameter matrix obtained in step 2, a matrix conversion relationship between the two-dimensional pixel coordinates and the three-dimensional world coordinates is established; a target workpiece image is collected, B feature points are selected on the workpiece surface, and three-dimensional coordinates of the workpiece surface feature points in the three-dimensional world coordinate system are calculated;

[0009] Step 4, according to the property that the distances of the workpiece surface feature points in the world coordinate system and the camera coordinate system are invariant, a linear equation group composed of a known matrix and an unknown matrix is constructed,

[0010] Step 5, the coefficients of the linear equation group are determined, and a target function is constructed, and the optimal solution of the equation after iteration is obtained through a least square optimization algorithm;

[0011] Step 6, the matrix representation of the feature points in the world coordinate system and the camera coordinate system after removing the centroid is established by combining singular value decomposition, and each auxiliary camera C i The pose relationship matrix relative to the main camera C1.

[0012] Further, in step 1, the N CMOS cameras are linearly arranged in sequence or arranged in a ring shape, or arranged in a disordered manner in space, as long as the adjacent two cameras are convergently installed and have a common field of view covering the workpiece.

[0013] Further, in step 1, the number N of CMOS cameras constituting the multi-view vision unit ranges from 3 to 8.

[0014] Further, in step 2, the specific calculation process of the checkerboard calibration is as follows:

[0015] Step 2.1, a pinhole imaging model of the camera is constructed, and a pinhole imaging principle is used to establish a world coordinate system O w -X w Y w Z w , a camera coordinate system O c -X c Y c Z c The homogeneous matrix conversion relationship between the image coordinate system o-xy and the pixel coordinate system o-uv is shown in formula (1):

[0016]

[0017] In formula (1), f is focal length, Z c is the longitudinal coordinate of the spatial point in the camera coordinate system, represents the intrinsic parameter matrix, R represents the rotation matrix, T represents the translation vector, (d x ,d y ) represents the physical size of the image coordinate system, and (u0, v0) represents the pixel coordinates of the origin of the image coordinate system.

[0018] Step 2.2, constructing a camera distortion model, solving the image distortion problem caused by factors such as manufacturing process and assembly error in the image plane imaging process, and the expression of the camera distortion model is as shown in formula (2):

[0019]

[0020] In formula (2), (x, y) is the ideal feature point coordinate, (x', y') is the actual detection point coordinate, k1 and k2 are the radial distortion coefficients, and p1 and p2 are the tangential distortion coefficients.

[0021] Step 2.3, constructing a convergent binocular camera imaging model with two adjacent cameras installed in a convergent manner, taking the left camera as the main camera and the right camera as the auxiliary camera in the two adjacent cameras; the origin of the left camera coordinate system is regarded as the origin of the world coordinate system, and the pose relationship between the left camera and the right camera is represented by formula (3):

[0022]

[0023] In formula (3), [X r ,Y r ,Z r ,1] T represents the left camera coordinate system in the convergent binocular camera imaging model, [X l ,Y l ,Z l ,1] T represents the right camera coordinate system in the convergent binocular camera imaging model, and the conversion matrix represents the rotation matrix of the external parameters of the right camera relative to the left camera in the convergent binocular camera imaging model, T l r =(t x ,t y ,t z ) T represents the translation vector of the external parameters of the right camera relative to the left camera in the convergent binocular camera imaging model.

[0024] Further, in step 3, the feature points on the workpiece surface are B feature points randomly selected in the common field of view of the multi-view vision unit, and B is not less than 3.

[0025] Further, in step 3, the number B of feature points selected from the workpiece surface is in the range of 3-10.

[0026] Further, in step 3, the specific calculation process of the three-dimensional coordinates of the feature points on the workpiece surface is as follows:

[0027] Step 3.1, the conversion relationship between the converging binocular camera coordinate system and the image coordinate system is shown in formula (4):

[0028]

[0029] In formula (4), is the left camera internal parameter matrix; is the right camera internal parameter matrix, f l is the left camera focal length, f r is the right camera focal length; (u l ,v l ) represents the pixel coordinates of the center point of the left camera image, (u r ,v r ) represents the pixel coordinates of the center point of the right camera image; Z l is the vertical coordinate of the space point in the left camera coordinate system, Z r is the vertical coordinate of the space point in the right camera coordinate system;

[0030] Step 3.2, in combination with the two-dimensional pixel coordinates (x l ,y l ) and (x r ,y r ) in the image coordinate system, the relationship matrix between the converging binocular cameras is shown in formula (5):

[0031]

[0032] In formula (5), (x l ,y l ) is the two-dimensional pixel coordinates of a point in the left camera image coordinate system, (x r ,y r ) is the two-dimensional pixel coordinates of a point in the right camera image coordinate system, f is the camera focal length, r is the rotation matrix parameter of the right camera relative to the left camera, t is the translation vector parameter of the right camera relative to the left camera, and Z is the vertical coordinate of the space point in the camera coordinate system;

[0033] Step 3.3, according to the converging binocular camera imaging model and in combination with the camera relationship matrix formula (5), the three-dimensional coordinates of the B non-coplanar feature points on the workpiece surface are obtained The space feature point and the i-th camera mapping point have a conversion relationship as where R iand T i respectively represent the rotation matrix and the translation vector of the external parameters of the i-th camera relative to the first main camera C1, where i≧3;

[0034] Step 3.4, obtaining the B feature points d n on the target workpiece by the B cameras to obtain a reference point P m , and determining m reference point groups by selecting m feature points on the surface of the workpiece, as shown in equation (6):

[0035]

[0036] In equation (6), and respectively represent the coordinates of the reference point in space and the i-th camera, and β mn represents the homogeneous barycentric coordinates of the n-th feature point in the m-th reference point group, which is uniquely determined by the feature point d n , n = 1, 2, 3…B.

[0037] Further, in step 3, the number of reference points m ranges from 3 to 10.

[0038] Further, in step 4, the specific calculation process of constructing the linear equation set is as follows:

[0039] Step 4.1, knowing the internal parameter matrix of the auxiliary camera C i , establishing the homogeneous relationship between the two-dimensional pixel coordinates and the three-dimensional camera coordinates of the auxiliary camera C i , as shown in equation (7):

[0040]

[0041] In equation (7), fx is the focal length component of the auxiliary camera C i in the X direction of the world coordinate system, f y is the focal length component of the auxiliary camera C i in the Y direction of the world coordinate system, (u0, v0) represents the pixel coordinates of the image center point in the auxiliary camera C i , (u m , v m ) represents the pixel coordinates of the corresponding point, β mn represents the homogeneous barycentric coordinates, represents the vertical coordinate of the space point in the camera coordinate system of the i-th CMOS camera, and s m represents the weighted value of the vertical coordinate of the space point in the camera coordinate system of the i-th CMOS camera.

[0042] Step 4.2, expanding and simplifying equation (7) to arrange it into the form of an equation set, as shown in equation (8):

[0043]

[0044] In formula (8), f x ,f y ,u0,v0 represent the internal parameters of the auxiliary camera C i , (u i ,v i ) represent the corresponding two-dimensional pixel coordinates of the reference point in the auxiliary camera C i , and are known quantities; i represents the three-dimensional camera coordinates of the feature point under the auxiliary camera C , which is an unknown quantity;

[0045] Step 4.3, formula (8) is expressed in linear equation form as D·X=0, D represents a known quantity matrix of 2m×12, and X represents an unknown quantity matrix of 12×1; further converted into formula (9) by right singular vector V i

[0046]

[0047] represents the coefficient of the linear equation group, and T represents the transpose matrix.

[0048] Further, in step 5, the specific calculation process of the iterative optimal solution of the linear equation group is as follows:

[0049] Step 5.1, combined with the spatial coordinates of B feature points n=1, 2, 3…Q and the property that the distance between feature points is equal in the world coordinate system and the camera coordinate system, the coefficient is determined as shown in formula (10):

[0050]

[0051] In formula (10), V [n] represents a vector composed of the nth feature point coordinates in V;

[0052] Step 5.2, formula (10) is expressed in linear equation form as Similarly, by using the equal distance between two feature points, the objective function is established as shown in formula (11):

[0053]

[0054] In formula (11), k, n represent the serial numbers of the feature points, and 1≤k<n≤B, is the objective function.

[0055] Step 5.3, formula (11) is solved to calculate​ Compared to Jacobian matrix J k,n As shown in formula (12):

[0056]

[0057] In formula (12), K i Let represent the matrix composed of the i-th right singular vector, where i is the subscript indicating the order, and its value is a positive integer between 1 and B. T represents the transpose matrix.

[0058] Step 5.4: The expression for synthesizing all Jacobian matrices of the B feature points that meet the conditions is shown in formula (13):

[0059]

[0060] In formula (13), ω represents the residual matrix, and J represents the Jacobian matrix of all combinations of distances between B feature points;

[0061] Step 5.5: Combine equations (12) and (13) to establish the solution. The incremental equation is calculated through multiple iterations to obtain the optimal solution, as shown in formula (14):

[0062]

[0063] In formula (14), μ is The coefficients are used for iterative updates.

[0064] Furthermore, in step 6, the least squares optimization algorithm is used to solve for... back, and All are known; establish the reference point with its centroid removed in the world coordinate system and the camera coordinate system of the i-th camera as a matrix representation, and use singular value decomposition to inversely obtain the rotation matrix and translation vector [R]. i |T i As shown in formula (15), the coordinates of the centroid of the reference point are... and

[0065] In formula (15), q c Let q be the matrix representation of the reference point in the camera coordinate system. w Let Q be the homography matrix of the reference point in the world coordinate system, [U∑V]=SVD(Q) be the singular value decomposition equation, and R be the reference point matrix in the world coordinate system. i Indicates auxiliary camera C i Relative to the rotation matrix of the main camera C1, T i Indicates auxiliary camera C iTranslation vector relative to the main camera C1.

[0066] The beneficial effects of the present application compared with the prior art are:

[0067] The present application firstly selects multiple CMOS cameras to form a multi-view vision unit, adjusts different viewing angles of the cameras, and ensures that the field of view covers the entire target part; secondly, collects and calibrates a sequence of checkerboard images under a common field of view, calculates the internal parameter matrix of each camera and the external parameter matrix between the converging binocular cameras; then collects the target workpiece image, calculates the three-dimensional coordinates of the surface feature points of the workpiece by establishing the matrix conversion relationship between the two-dimensional pixel coordinates and the three-dimensional world coordinates; thirdly, a linear equation set is constructed according to the equal distance property of the surface feature points in the world coordinate system and the camera coordinate system, and the optimal solution of the iterative equation is obtained by using the least square optimization algorithm; finally, the matrix representation of the feature points in the world coordinate system and the camera coordinate system is established by removing the centroid through singular value decomposition, and the pose relationship matrix of the auxiliary camera relative to the main camera is reversely calculated. The method fully considers the actual problems such as insufficient field of view and regional occlusion in the imaging process of large and complex workpieces, can calibrate multiple views through the construction of a multi-view vision unit, and further improve the field of view range and calibration accuracy, and has strong practical value. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 It is a flowchart of the multi-view calibration method for large and complex workpieces in the method embodiment of the present application;

[0069] Figure 2 It is a schematic diagram of the camera pinhole imaging and distortion imaging model described in the embodiment of the present application;

[0070] Figure 3 It is a schematic diagram of the converging binocular camera imaging model described in the embodiment of the present application;

[0071] Figure 4 It is a schematic diagram of the multi-view camera imaging model described in the embodiment of the present application;

[0072] Figure 5 It is a schematic diagram of the reverse calibration of the external parameter matrix. DETAILED DESCRIPTION

[0073] The preferred embodiments of the present application will be specifically described below in combination with the drawings, wherein the drawings constitute a part of the present application, and are used together with the embodiments of the present application to explain the principles of the present application, and are not used to limit the scope of the present application.

[0074] Embodiment 1:

[0075] In this embodiment, three CMOS cameras form a multi-view vision unit, and four non-coplanar feature points on the workpiece surface are selected for illustration, please refer toFigure 1 The embodiment of the present application provides a flowchart of a multi-view calibration method for a large and complex workpiece, which comprises the following steps.

[0076] Step 1, three CMOS cameras are selected to form a multi-view vision unit, and the field of view range is ensured to cover the entire target workpiece by adjusting the different view angles of the cameras.

[0077] In particular, the three CMOS cameras selected in the embodiment are arranged in sequence to form a multi-view vision unit in a converging installation mode of every two cameras, and the multi-view vision unit has a common field of view between the cameras, and the field of view range covers the surface of the workpiece. It should be noted that the common field of view only needs to cover part of the surface of the workpiece, and does not need to cover the entire surface. Of course, the larger the coverage area is, the better. The local coverage mode can greatly expand the overall field of view of the multi-view vision unit.

[0078] Step 2, under the common field of view, a sequence of checkerboard images is collected, and by calibrating the checkerboard, the internal parameter matrix of each camera and the external parameter matrix between the main camera C1 and the adjacent auxiliary camera C2 are calculated. The specific method is as follows:

[0079] a, a pinhole imaging model of the camera is constructed, as shown in the following formula (1): Figure 2 The pinhole imaging principle is used to establish a world coordinate system O w -X w Y w Z w , a camera coordinate system O c -X c Y c Z c , a image coordinate system o-xy and a pixel coordinate system o-uv are shown in formula (1):

[0080]

[0081] In formula (1), f is the focal length, Z c is the vertical coordinate of the space point in the camera coordinate system, represents the internal parameter matrix, R represents the rotation matrix, T represents the translation vector, (d x , d y ) represents the physical size of the image coordinate system, and (u0, v0) represents the pixel coordinates of the origin of the image coordinate system.

[0082] b, a camera distortion model is constructed, as shown in the following formula (2): Figure 2 In the image plane imaging process, the image distortion problem caused by factors such as manufacturing process and assembly error is solved, and the camera distortion model expression is shown in formula (2):

[0083]

[0084] In formula (2), (x, y) is the ideal feature point coordinate, (x', y') is the actual detection point coordinate, k1, k2 are radial distortion coefficients, p1, p2 are tangential distortion coefficients;

[0085] c. The converging binocular camera imaging model of the main camera C1 and the adjacent auxiliary camera C2 is constructed as shown in the following formula (3). Figure 3 The left camera is taken as the main camera, the camera coordinate system origin is taken as the world coordinate system origin, and the pose relationship between the main camera C1 and the adjacent auxiliary camera C2 is represented by formula (3).

[0086]

[0087] In formula (3), [X r ,Y r ,Z r ,1] T and [X l ,Y l ,Z l ,1] T represent the left and right camera coordinate systems in the converging binocular camera imaging model, respectively. R l r =(t x ,t y ,t z ) T represents the rotation matrix of the right camera relative to the left (main) camera in the converging binocular camera imaging model.

[0088] Step 3: According to the camera parameter matrix obtained in step 2, the matrix conversion relationship between the two-dimensional pixel coordinates and the three-dimensional world coordinates is established, the target workpiece image is collected, B feature points are selected on the workpiece surface in the common field of view, and the three-dimensional coordinates of the workpiece surface feature points in the three-dimensional world coordinate system are calculated.

[0089] In particular, the workpiece surface feature points in the embodiment need to be selected arbitrarily in the common field of view of the multi-view vision unit, as shown in the following formula (4). Figure 4

[0090] a. The conversion relationship between the converging binocular camera coordinate system and the image coordinate system is shown in formula (4):

[0091]

[0092] In formula (4), A and A are the left and right camera internal parameter matrices in the converging binocular camera imaging model, respectively, f is the focal length, and f l ​f is the focal length of the left camera r f is the focal length of the right camera; (u, v) represents the pixel coordinates of the image center point, (u l ,v l ) represents the pixel coordinates of the image center point of the left camera, (u r ,v r ) represents the pixel coordinates of the image center point of the right camera; Z is the vertical coordinate of the space point in the camera coordinate system, Z l is the vertical coordinate of the space point in the left camera coordinate system, and Z r is the vertical coordinate of the space point in the right camera coordinate system.

[0093] b. Combined with the two-dimensional pixel coordinates (x l ,y l ) and (x r ,y r ) in the image coordinate system, the inter-camera relationship matrix of the converging binocular camera is shown in formula (5):

[0094]

[0095] In formula (5), (x l ,y l ) is the two-dimensional pixel coordinates of a point in the left camera image coordinate system, (x r ,y r ) is the two-dimensional pixel coordinates of a point in the right camera image coordinate system, f is the focal length of the camera, r is the rotation matrix parameter of the right camera relative to the left camera, t is the translation vector parameter of the right camera relative to the left camera, and Z is the vertical coordinate of the space point in the camera coordinate system.

[0096] c. From the imaging model of the converging binocular camera, combined with the inter-camera relationship matrix formula (5), the three-dimensional coordinates of the four non-coplanar feature points on the workpiece surface space feature points and the camera mapping points of the auxiliary camera C3 have a conversion relationship as where R3 and T3 represent the rotation matrix and translation vector of the third camera relative to the first main camera C1, respectively.

[0097] d. A reference point P n is obtained by weighting the four feature points d m on the target workpiece, and m reference point groups are determined by selecting m feature points on the workpiece surface, as shown in formula (6).

[0098]

[0099] In formula (6), and respectively, β mn denote the homogeneous barycentric coordinates, d n is uniquely determined, n = 1, 2, 3, 4.

[0100] Step 4, according to the property that the distance between the feature points of the workpiece in the world coordinate system and the camera coordinate system is invariant, a linear equation group composed of known quantity matrix and unknown quantity matrix is constructed,

[0101] a, the internal parameter matrix of the auxiliary camera C3 is known, and the homogeneous relationship between the corresponding points of the two-dimensional pixel coordinates of the auxiliary camera C3 and the three-dimensional camera coordinates is established as shown in formula (7):

[0102]

[0103] In formula (7), f is the focal length, fx is the component of the focal length of the auxiliary camera C3 in the X direction of the world coordinate system, f y is the component of the focal length of the auxiliary camera C3 in the Y direction of the world coordinate system; (u0, v0) represents the pixel coordinates of the image center point in the auxiliary camera C3, (u m ,v m ) represents the pixel coordinates of the corresponding points, β mn denotes the homogeneous barycentric coordinates, denotes the vertical coordinate of the space point in the camera coordinate system of the third CMOS camera, s m denotes the weighted value of the vertical coordinate of the space point in the third camera coordinate system;

[0104] b, after expanding and simplifying formula (7), it is arranged into the form of an equation group as shown in formula (8):

[0105]

[0106] In formula (8), f x ,f y , u0, v0 represent the camera internal parameters of the auxiliary camera C3, (u i ,v i ) represents the corresponding two-dimensional pixel coordinates of the reference point in the auxiliary camera C3, and both are known quantities. denotes the three-dimensional camera coordinates of the feature point under the auxiliary camera C3, which is an unknown quantity;

[0107] c, formula (8) is expressed in the form of a linear equation as D·X = 0, D represents a known quantity matrix of 2m×12, and X represents an unknown quantity matrix of 12×1. Further, by using the right singular vector V i , formula (9) is converted.

[0108]

[0109] represents the coefficient of the linear equation group, T represents the transpose matrix.

[0110] Step 5, determining the coefficient of the linear equation group and constructing the objective function, and obtaining the optimal solution of the equation after iteration through the least square optimization algorithm;

[0111] a, combining the spatial coordinates of the four feature points n=1, 2, 3, 4 and the property that the distance between the feature points is equal in the world coordinate system and the camera coordinate system, the coefficient As shown in formula (10):

[0112]

[0113] In formula (10), V [n] represents the vector composed of the nth feature point coordinate in V;

[0114] b, formula (10) is expressed in the form of linear equation as Similarly, by using the equal distance between the two feature points, the objective function is established as shown in formula (11):

[0115]

[0116] In formula (11), k and n represent the serial number of the feature points, and 1≤k is the objective function;

[0117] c, solving formula (11) to calculate relative to the Jacobian matrix J k,n , as shown in formula (12):

[0118]

[0119] d, further, the expression of all Jacobian matrices synthesized after meeting the conditions of the four feature points is shown in formula (13):

[0120]

[0121] In formula (13), ω represents the residual matrix, and J represents the Jacobian matrix of all combinations of the distances between the four feature points;

[0122] e, combining formula (12) and formula (13), the incremental equation of is established, and the optimal solution is obtained through multiple iterations, and the expression is shown in formula (14):

[0123]

[0124] In formula (14), μ is a coefficient of formula (14) for iterative updating.

[0125] Step 6, singular value decomposition is used to establish the matrix representation of the feature points without the centroid in the world coordinate system and the camera coordinate system, and the pose relationship matrix of the auxiliary camera C3 relative to the main camera C1 is reversely calculated.

[0126] In particular, the least square optimization algorithm is used to solve After that, and are known.The matrix representation of the reference points without the centroid in the world coordinate system and the camera coordinate system of the auxiliary camera C3 is established, as shown in formula (15), and the rotation matrix and the translation vector [R3|T3] are reversely solved by using singular value decomposition, as shown below. Figure 5 The centroid coordinates of the reference points are and

[0127]

[0128] In formula (15), q c is the matrix representation of the reference points in the camera coordinate system, q w is the matrix representation of the reference points in the world coordinate system, Q is a homography matrix, [U∑V]=SVD(Q) is a singular value decomposition equation, R3 represents the rotation matrix of the auxiliary camera C3 relative to the main camera C1, and T3 represents the translation vector of the auxiliary camera C3 relative to the main camera C1.

[0129] Embodiment 2: A multi-view vision unit is formed by using four CMOS cameras, four feature points not on the same plane are selected on the surface of a workpiece, the external parameter matrix between the third camera and the main camera C1 is calculated in the manner of embodiment 1, and then the external parameter matrix between the fourth CMOS camera (auxiliary camera C4) and the main camera C1 is calculated in the same manner as that of embodiment 1. By analogy, the external parameter matrix between the Nth CMOS camera (auxiliary camera C4) and the main camera C1 can be calculated, so that the multi-view joint calibration is completed.

[0130] The main innovations of the method are as follows: 1) a multi-view joint calibration method for large and complex workpieces is established in the field of computer vision joint calibration; 2) the multi-view vision unit is fully utilized to improve the joint calibration precision and algorithm reliability due to the large imaging range and high calibration precision; 3) the function of reversely calibrating the pose relationship between the auxiliary camera and the main camera is realized by constructing linear equations, objective functions, least square optimization and singular value decomposition algorithms.

[0131] The above examples are only used for illustrating the design idea and characteristics of the present application, and the purpose is to enable the person skilled in the art to understand the present application and to implement it, and the protection scope of the present application is not limited to the above examples. Therefore, any equivalent changes or modifications made according to the disclosed principles and design ideas of the present application are within the protection scope of the present application.

Claims

1. A multi-view joint calibration method for large complex workpieces, characterized in that, Comprise the following steps: Step 1, selection N The multi-view vision unit is composed of CMOS cameras, and the different viewing angles of the cameras are adjusted to ensure that the field of view covers the entire target workpiece, N Greater than or equal to 3, in the form of convergent installation of every two adjacent cameras, to form a multi-view vision unit, and the multi-view vision unit has a common field of view covering the surface of the target workpiece. Step 2, define the first camera as the main camera C 1, the rest of the cameras are arranged in order, and are auxiliary cameras C i , i The value is between 2 and N , wherein the auxiliary cameras C 2 are convergently installed with the main camera C 1, and every two adjacent auxiliary cameras are also convergently installed; a sequence of checkerboard images is collected under a common field of view, the checkerboard is calibrated, and a camera parameter matrix of the multi-view vision unit is calculated, the camera parameter matrix including an intrinsic parameter matrix of each camera and an extrinsic parameter matrix between every two adjacent cameras; Step 3, according to the camera parameter matrix obtained in step 2, the matrix conversion relationship between two-dimensional pixel coordinates and three-dimensional world coordinates is established; the target workpiece image is collected, a feature point is selected on the workpiece surface, and the three-dimensional coordinates of the feature point on the workpiece surface in the three-dimensional world coordinate system are calculated B . Step 4, according to the nature of the workpiece surface feature points in the world coordinate system and the camera coordinate system distance invariable, construct the linear equations with known quantity matrix and unknown quantity matrix, Step 5, determine the coefficient of linear equations, and construct the objective function, get the optimal solution of the equation after iteration through the least square optimization algorithm; Step 6, combine singular value decomposition to establish the matrix representation of the feature points removed from the centroid in the world coordinate system and the camera coordinate system, and inversely calculate each auxiliary camera C i with respect to the pose relationship matrix of the main camera C 1; In step 4, the specific calculation process of constructing linear equations is as follows: Step 4.1, known secondary camera C i matrix of intrinsic parameters of the secondary camera C i The homogeneous relationship between the corresponding points of the two-dimensional pixel coordinates and the three-dimensional camera coordinates of the secondary camera is shown in formula (7): Equation (7) In formula (7), Fx auxiliary camera C i The component of the focal length in the X direction in the world coordinate system. f y auxiliary camera C i The component of the focal length in the Y direction in the world coordinate system, ( u 0, v 0) indicates auxiliary camera C i The pixel coordinates of the center point of the image, u m , v m () represents the pixel coordinates of the corresponding point. β mn Represents the coordinates of the homogeneous barycenter. This indicates that the spatial point is at the 1st position. i The ordinate of a CMOS camera in the camera coordinate system. s m Representing the spatial point at the th i The weighted value of the ordinate in the camera coordinate system of a CMOS camera; Step 4.2, expand and simplify formula (7) and arrange it into the form of equation group, as shown in formula (8): Formula (8) In formula (8), f x , f y , u 0, v 0 indicates auxiliary camera C i The intrinsic parameters, ( u i , v i () indicates that the reference point is at the auxiliary camera. C i The corresponding two-dimensional pixel coordinates are all known quantities; Indicates feature points in the auxiliary camera C i The 3D camera coordinates are unknowns. Step 4.

3. Equation (8) is expressed in linear equation form as D . X =0, D represents a known matrix of 2 m x 12, X represents an unknown matrix of 12 x 1; further expressed in right singular vectors V i into equation (9) Formula (9) denotes the coefficients of the linear equation system, T denotes the transposed matrix.

2. The multi-view joint calibration method for large complex workpieces according to claim 1, wherein: In step 1, N The CMOS cameras are linearly arranged in sequence or in a ring shape.

3. The multi-view joint calibration method for large complex workpieces of claim 1, wherein: In step 2, the specific calculation process of the chessboard calibration is as follows: Step 2.1, Constructing camera pinhole imaging model, using pinhole imaging principle to establish world coordinate system O w - X w Y w Z w , camera coordinate system O c - X c Y c Z c , image coordinate system o - Xy and pixel coordinate system o - Uv The homogeneous matrix conversion relationship between them is shown in formula (1): Equation (1) In formula (1), f Focal length Z c Let be the ordinate of the point in the camera coordinate system. Represents the intrinsic parameter matrix, R Represents the rotation matrix. T Represents the translation vector, ( d x , d y ) represents the physical dimensions of the image coordinate system, ( u 0, v 0) represents the pixel coordinates of the origin of the image coordinate system; Step 2.2, construct the camera distortion model, solve the image distortion problem caused by manufacturing process and assembly error factors in the image plane imaging process, and the expression of the camera distortion model is as shown in formula (2): Formula (2) In formula (2), ( x , y ) represents the coordinates of the ideal feature point. x’ , y’ ( ) represents the actual coordinates of the detection point. k 1. k 2 represents the radial distortion coefficient. p 1. p 2 represents the tangential distortion coefficient; Step 2.3, build a convergent binocular camera imaging model with two adjacent cameras installed in a convergent manner, take the left camera as the main camera and the right camera as the auxiliary camera in the two adjacent cameras; The origin of the left camera coordinate system is regarded as the origin of the world coordinate system, and the pose relationship between the left camera and the right camera is represented by formula (3): Formula (3) In formula (3), [Mathematical Figure 3] X r , Y r , Z r , 1] T represents the left camera coordinate system in the converging binocular camera imaging model, [Mathematical Figure 4] X l , Y l , Z l , 1] T represents the right camera coordinate system in the converging binocular camera imaging model, and the conversion matrix , represents the rotation matrix of the right camera relative to the left camera in the converging binocular camera imaging model, represents the translation vector of the right camera relative to the left camera in the converging binocular camera imaging model.

4. The multi-view joint calibration method for large complex workpieces of claim 1, wherein: In step 3, the feature points of the workpiece surface are randomly selected within the common field of view of the multi-vision unit B not coplanar, B not less than 3.

5. The multi-view joint calibration method for large complex workpieces according to claim 3, wherein: In step 3, the specific calculation process of the three-dimensional coordinates of the workpiece surface feature points is as follows: Step 3.1, the conversion relationship between the convergent binocular camera coordinate system and the image coordinate system is as shown in formula (4): Equation (4) In formula (4), is a left camera intrinsic parameter matrix; is a right camera intrinsic parameter matrix, f l is a left camera focal length, f r is a right camera focal length; u l , v l represents a left camera image center point pixel coordinate, u r , v r represents a right camera image center point pixel coordinate; Z l is a vertical coordinate of a space point in a left camera coordinate system, Z r is a vertical coordinate of a space point in a right camera coordinate system; Step 3.2, combine the two-dimensional pixel coordinates of the image coordinate system in which the image is located ( x l , y l ) and ( x r , y r ), the convergent binocular camera interrelation matrix is shown in formula (5): Formula (5) In formula (5), (x, y) is the two-dimensional pixel coordinate of a point in the left camera image coordinate system, (x', y') is the two-dimensional pixel coordinate of a point in the right camera image coordinate system, f is the focal length of the camera, R is the rotation matrix parameter of the right camera relative to the left camera, t is the translation vector parameter of the right camera relative to the left camera, and z is the longitudinal coordinate of the space point in the camera coordinate system. x l , y l ) for the two-dimensional pixel coordinate of a point in the right camera image coordinate system, x r , y r ) for the two-dimensional pixel coordinate of a point in the right camera image coordinate system, f for the focal length of the camera, r for the rotation matrix parameter of the right camera relative to the left camera, t for the translation vector parameter of the right camera relative to the left camera, Z for the longitudinal coordinate of the space point in the camera coordinate system. Step 3.3: Based on the convergent binocular camera imaging model and the inter-camera relationship matrix formula (5), the workpiece surface is obtained. B Three-dimensional coordinates of non-coplanar feature points Spatial feature points and the i Taiwan camera mapping point There is a conversion relationship between them. ,in R i and T i They represent the first i The second camera relative to the first main camera C 1. Extrinsic parameter rotation matrix and extrinsic parameter translation vector, at this point i ≥3; Step 3.4: Passing through the target workpiece B Feature points d n We obtain a reference point by weighting. P m The feature points on the workpiece surface are determined by selecting m times. m A set of reference points, as shown in formula (6): Equation (6) In equation (6), and denote the coordinates of the reference point in the space and the first i camera, β mn denote the homogeneous barycentric coordinates of the i-th feature point in the first m group of reference points, determined by the feature point n n d n = 1, 2, 3… B .​​ 6. The multi-view joint calibration method for large complex workpieces of claim 1, wherein: In step 5, the specific calculation process of the iterative optimal solution of the linear equations is as follows: Step 5.1, combining B the feature point space coordinates and the property that the distances between the feature points are equal in the world coordinate system and the camera coordinate system, determining the coefficients Phi i As shown in equation (10): Equation (10) In formula (10), V [n] denotes V the vector composed of the coordinates of the n characteristic points in the middle of the Step 5.2, Equation (10) is expressed in linear equation form as L . Phi =0, Phi [ Phi 11 , Phi 12 , Phi 13 … Phi 1B , Phi 22 , Phi 23 , Phi 24 , Phi 33 , Phi 34 , Phi 4 B ] T ; Similarly, the equal distance between the feature points is used to establish the objective function as shown in formula (11): Equation (11) In Equation (11), k , n denotes a feature point serial number, and , is an objective function; Step 5.3, solve equation (11) to calculate E k,n ( Phi ) the Jacobian matrix of Phi J k,n as shown in equation (12):​ Formula (12) In equation (12) K i denotes a matrix composed of the first i right singular vectors, where i is an index denoting the order, and takes on positive integer values from 1 to B T denotes the transpose matrix. Step 5.4, the expression of all Jacobian matrices after being synthesized for the feature points satisfying the condition is shown in equation (13): B The expression of all Jacobian matrices after being synthesized for the feature points satisfying the condition is shown in equation (13): Equation (13) In Equation (13), Omega denotes the residual matrix, J denotes B the Jacobian matrix of all combinations of distances between the k feature points; Step 5.5, the incremental equation of formula (12) and formula (13) are established to obtain the optimal solution through multiple iterations, and the expression is shown in formula (14): Phi ​ Equation (14) In formula (14), Mu for Phi The coefficients are used for iterative updates.

7. The multi-view joint calibration method for large complex workpieces according to claim 6, characterized in that: In step 6, the least square optimization algorithm is used to solve Phi After, And are known; the matrix representation of the reference point removing the centroid in the world coordinate system and the camera coordinate system of the first i camera is used to inversely solve the rotation matrix and the translation vector R i | T i , as shown in equation (15), the reference point centroid coordinates are And ; Equation (15) In equation (15), q c is the matrix representation of the reference point in the camera coordinate system, q w is the matrix representation of the reference point in the world coordinate system, Q is the homography matrix, U∑V = SVD is the singular value decomposition equation, Q ) is the singular value decomposition equation, R i denotes the secondary camera C i the rotation matrix of the secondary camera C 1 with respect to the primary camera T i denotes the secondary camera C i the translation vector of the secondary camera C1 with respect to the primary camera.

8. The multi-view joint calibration method for large complex workpieces according to any one of claims 1-7, characterized in that: The number of feature points selected on the workpiece surface B The value range is 3-10.

9. The multi-view joint calibration method for large complex workpieces according to any one of claims 1-7, characterized in that: Number of CMOS cameras constituting a multi-view vision unit N The value range is 3-8.

Images