A joint calibration method and device for 2D area array camera and line laser 3D sensor

Abstract

The present invention discloses a joint calibration method of a 2D array camera and a line laser 3D sensor. The method comprises: placing a target with no less than three cylindrical features at a common measurement position of the line laser 3D sensor and the 2D array camera, the 2D array camera acquiring images of three groups of intersection lines of laser surfaces and cylinders, and then acquiring image coordinates of three groups of laser light strips respectively; calculating the 3D coordinates of the light strip points in the 2D array camera coordinate system according to the 3D coordinates thereof, and respectively calculating the coordinates of the center positions of three ellipses by using an ellipse fitting algorithm; directly acquiring the 3D coordinates of the three groups of laser light strips in the 3D sensor coordinate system from the line laser 3D sensor, and respectively calculating the coordinates of the center positions of three ellipses by using an ellipse fitting algorithm; calculating the conversion distance between the two coordinate systems by using the 3D coordinates of the 2D array camera and the 3D coordinates of the line laser 3D sensor, and completing the joint calibration.

Description

Technical Field

[0001] The invention relates to a 2D area array camera and a line laser 3D sensor joint calibration method and device, belonging to the technical field of visual 3D measurement. Background Art

[0002] Line laser-based 3D sensors have become an important type of sensor on the market, and there are already quite a few mature products. Line laser 3D sensors project line lasers onto the surface of the object to be measured, which are modulated by the surface of the object to form light strips. After being collected by the image sensor and processed by calculation, the 3D point coordinates of the light strips on the surface of the object can be directly output. Among them, the measurement coordinate system is defined by the sensor itself, and its specific position and internal parameters of the 3D sensor are generally not public, which brings inconvenience to the need to build multi-type, heterogeneous visual measurement systems.

[0003] At present, in a multi-eye vision measurement system that uses two or more industrial cameras, the measurement results of each individual visual sensor need to be unified into a 3D coordinate system. Therefore, it is necessary to know the relative position relationship between multiple visual sensors, which is represented by the rotation matrix R and the translation vector t. The process of obtaining the R and t matrices is called the external parameter calibration between binocular or multi-eye industrial cameras. Generally, the internal parameters of each industrial camera are used to complete the calibration with the help of two-dimensional plane targets, stereo targets, etc. However, some integrated 3D sensors generally only output three-dimensional coordinates in their own defined coordinate system, and do not open their internal parameters and process images taken by the camera to the outside world. Therefore, the existing traditional methods are used to perform external parameter calibration between multiple sensors. For example, in a combined measurement system using a line laser 3D sensor and a 2D array camera, the output results of the line laser 3D sensor need to be imported into the coordinate system of other 2D array cameras. In this case, the relative position relationship between the 2D array camera and the 3D sensor needs to be obtained. However, since the 3D sensor does not disclose internal parameters and does not output image information, the traditional binocular calibration method cannot be used. Summary of the invention

[0004] The object of the present invention is to provide a method and device for joint calibration of a 2D area array camera and a line laser 3D sensor, so as to solve the defects of the prior art.

[0005] A joint calibration method of a 2D area array camera and a line laser 3D sensor, the method comprising:

[0006] The line laser of the line laser 3D sensor and the 2D array camera form a line structured light vision model, and the line structured light vision model calibration is completed using the two-dimensional plane target method;

[0007] Establish the origin of the 3D coordinate system of the line laser 3D sensor at the optical center of the 2D array camera, and obtain the 3D coordinates of the light strip points on the 2D array camera;

[0008] Place a target with no less than three cylindrical features at the common measurement position of the line laser 3D sensor and the 2D array camera. The 2D array camera obtains images of three groups of intersections of the laser surface and the cylinder, and then obtains the image coordinates of the three groups of laser light strips respectively.

[0009] The 3D coordinates of the light strip points in the 2D array camera coordinate system are calculated based on the 3D coordinates, and the coordinates of the center positions of the three ellipses are calculated using the ellipse fitting algorithm.

[0010] The 3D coordinates of the three groups of laser light strips in the 3D sensor coordinate system are directly obtained from the line laser 3D sensor, and the coordinates of the center positions of the three ellipses are calculated respectively using the ellipse fitting algorithm;

[0011] Using the 3D coordinates of the 2D area array camera and the 3D coordinates of the line laser 3D sensor, the conversion distance between the two coordinate systems is calculated to complete the joint calibration.

[0012] Furthermore, the method for obtaining the 3D coordinates of the light stripe points on the 2D area array camera includes:

[0013] The X-axis of the line structured light vision model is the horizontal axis of the photosensitive surface, and the Y-axis is the vertical axis of the photosensitive surface;

[0014] The internal parameters of the 2D array camera are obtained as follows:

[0015]

[0016] Then the relationship between the image coordinates of a light strip point on the 2D array camera and its 3D coordinates is:

[0017] s·[uv 1] T =K·[xyz] T

[0018] After the line structured light vision model is calibrated, the coordinates of the laser surface under the 2D array camera are:

[0019] [abcd]·[xyz 1] T =0

[0020] From the above equation, the 3D coordinates of a light strip point on the 2D array camera can be obtained as:

[0021]

[0022] Furthermore, the cylindrical features are installed in parallel on a plane, any three of the central axes of the multiple cylinders are not coplanar, and the relative positional relationship between the cylinders remains unchanged after installation.

[0023] Furthermore, the target is placed at least once to complete the calibration.

[0024] Furthermore, the target is placed so as to obtain the intersection line between the laser plane and the three cylinders, and is simultaneously within the acquisition range of the 3D sensor and the 2D area array camera.

[0025] A 2D array camera and line laser 3D sensor joint calibration device, comprising no less than 3 cylindrical calibration targets, a line laser 3D sensor and a 2D array camera;

[0026] The central axes of each cylinder of the target are parallel to each other, and the central axes of any three cylinders are not in the same plane;

[0027] During calibration, the multi-cylinder target is placed in the common measurement area of ​​the line laser 3D sensor and the 2D array camera. The line laser of the line laser 3D sensor intersects with the cylinder to form three arc-shaped laser light strips. The 2D array camera and the line laser constitute a line structured light vision model, and the calibration is completed based on the planar target method.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention proposes a joint calibration method and device for a 2D area array camera and a line laser 3D sensor based on multi-cylinder targets. Compared with the traditional method, firstly, the measurement results of the 3D sensor can be directly used, without the need to open its internal parameters or obtain the two-dimensional images captured by it; secondly, there is no need to make particularly precise targets, only the cylindricity of the cylinder is required, the machining difficulty is low using machining, and there is no need to obtain the coordinate values ​​of the target feature points in advance; thirdly, without considering random errors, the joint calibration can be completed by placing and shooting once, without relying on multiple placements. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the device of the present invention;

[0031] In the figure: 1-line laser 3D sensor; 2-2D array camera; 3-target; 4-laser surface; 5-elliptical arc segment formed by the intersection of the laser surface and the target cylindrical feature. DETAILED DESCRIPTION

[0032] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0033] The present invention discloses a joint calibration method of a 2D area array camera and a line laser 3D sensor, the method comprising:

[0034] (1) A target is prepared that includes no less than three cylindrical features, wherein the cylindrical features are installed in parallel on a plane, and any three of the central axes of the multiple cylinders are not coplanar, and after installation, the relative positional relationship between the cylinders remains unchanged;

[0035] (2) The positions of the line laser 3D sensor and the 2D array camera are fixed, and their measurement areas overlap;

[0036] (3) The line laser of the line laser 3D sensor and the 2D array camera form a separate line structured light vision model. The line structured light vision model is calibrated using a two-dimensional plane target method. The origin of the 3D coordinate system is established at the optical center of the 2D array camera, the X-axis is the horizontal axis of the photosensitive surface, and the Y-axis is the vertical axis of the photosensitive surface.

[0037] The internal parameters of the 2D array camera are obtained as follows:

[0038]

[0039] Then the relationship between the image coordinates of a light strip point on the 2D array camera and its 3D coordinates is:

[0040] s·[uv 1] T =K·[xyz] T

[0041] After the line structured light vision model is calibrated, the coordinates of the laser surface under the 2D array camera are:

[0042] [abcd]·[xyz 1] T =0

[0043] From the above equation, the 3D coordinates of a light strip point on the 2D array camera can be obtained as:

[0044] x=z·(u-u0) / fx

[0045] y=z·(v-v0) / fy

[0046] z=-1·[a·(u-u0) / fx+b·(v-v0) / fy+c] -1

[0047] Where: fx is the x-axis component of the camera focal length, fy is the y-axis component of the camera focal length, u0 is the u-axis coordinate of the camera image origin, v0 is the v-axis coordinate of the camera image origin, u is the u-axis coordinate of the camera image pixel, v is the v-axis coordinate of the camera image pixel, x is the x-axis coordinate of the measured point in the camera space coordinate system, y is the y-axis coordinate of the measured point in the camera space coordinate system, and z is the z-axis coordinate of the measured point in the camera space coordinate system; a, b, c, d are the laser surface coordinate coefficients; T represents vector transpose, and s is the scale factor.

[0048] (4) The target is placed at the common measurement position of the line laser 3D sensor and the 2D array camera. The 2D array camera obtains the image of the intersection of the three laser surfaces and the cylinder, and obtains the image coordinates of the three groups of laser light strips respectively. The above formula is used to calculate the 3D coordinates in the 2D array camera coordinate system, and the ellipse fitting algorithm is used to calculate the coordinates of the three ellipse center positions respectively, which are set as C1, C2, and C3. The coordinates are in the form of column vectors. If there are more than 3 cylinders, the center position coordinates are set as C1, C2, C3...Cn

[0049] (5) Directly obtain the 3D coordinates of the three groups of laser light strips in the 3D sensor coordinate system from the line laser 3D sensor, and use the ellipse fitting algorithm to calculate the coordinates of the center positions of the three ellipses, namely D1, D2, and D3, in the form of column vectors. If there are more than 3 cylinders, the coordinates of the center positions are set to D1, D2, D3, ... Dn

[0050] (6) Assume that the transformation matrix from the 3D sensor to the 2D array camera is R, t, where R is the rotation matrix and t is the translation vector. Then, R and t can be calculated by the following formula using the least squares method to complete the joint calibration.

[0051] [C1 C2 C3...Cn]=R[D1 D2 D3...Dn]+t·[1 1 1...1].

[0052] like Figure 1 The invention also discloses a 2D array camera and a line laser 3D sensor joint calibration device, which also includes:

[0053] The design uses a calibration target of no less than 3 cylinders;

[0054] The target only needs to fix the three cylinders processed by lathe on a flat bottom plate, and there is no need to obtain the relative position relationship of the three cylinders in advance;

[0055] The central axes of each cylinder of the target are parallel to each other, but the central axes of any three cylinders are not in the same plane.

[0056] During calibration, the multi-cylinder target is placed in the common measurement area of ​​the line laser 3D sensor and the 2D array camera. The line laser of the line laser 3D sensor intersects with the cylinder to form three arc-shaped laser light strips. The 2D array camera and the line laser constitute a line structured light vision model, which can be calibrated using a planar target-based method. The 3D point coordinates of the three light strips in the 2D array camera coordinates can be obtained through the line structured light vision model. The three light strips are all elliptical arc segments on their laser planes. The elliptical curve fitting algorithm can be used to obtain the coordinates of the ellipse center, which is the intersection coordinates of the laser plane and the central axis of the three cylinders, and is expressed in the 2D array camera coordinate system. At the same time, the 3D point coordinates of the three light strips in the 3D sensor coordinate system can be directly obtained by the sensor. The same method is used to obtain the intersection coordinates of the laser plane and the central axis of the three cylinders, and it is expressed in the 3D sensor coordinate system. Since the intersection point of the laser plane and the central axis of the three cylinders is fixed and unique, the coordinate values ​​of the three intersection points in the 2D area array camera coordinate system and the 3D sensor coordinate system can be used to calculate the conversion distance between the two coordinate systems and complete the joint calibration.

[0057] In this embodiment, the calibration can be completed by placing the target at least once, and the influence of random errors can be reduced by placing the target multiple times;

[0058] The target placement only needs to ensure that the intersection line between the laser plane and the three cylinders can be obtained, and at the same time it is within the acquisition range of the 3D sensor and the 2D array camera. There is no requirement for the target placement angle.

[0059] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A joint calibration method of a 2D array camera and a line laser 3D sensor, characterized in that: The method comprises: The line laser of the line laser 3D sensor and the 2D array camera form a line structured light vision model, and the line structured light vision model calibration is completed using the two-dimensional plane target method; Based on the calibration, the origin of the 3D coordinate system of the line laser 3D sensor is established at the optical center of the 2D array camera, and the 3D coordinates of the light strip points on the 2D array camera are obtained; Place a target with no less than three cylindrical features at the common measurement position of the line laser 3D sensor and the 2D array camera. The 2D array camera obtains images of three groups of intersections of the laser surface and the cylinder, and then obtains the image coordinates of the three groups of laser light strips respectively. The 3D coordinates of the light strip points in the 2D array camera coordinate system are calculated based on the 3D coordinates, and the coordinates of the center positions of the three ellipses are calculated using the ellipse fitting algorithm. The 3D coordinates of the three groups of laser light strips in the 3D sensor coordinate system are directly obtained from the line laser 3D sensor, and the coordinates of the center positions of the three ellipses are calculated respectively using the ellipse fitting algorithm; Using the 3D coordinates of the 2D area array camera and the 3D coordinates of the line laser 3D sensor, the conversion distance between the two coordinate systems is calculated to complete the joint calibration.

2. The method for joint calibration of a 2D area array camera and a line laser 3D sensor according to claim 1, characterized in that: The method for obtaining the 3D coordinates of the light stripe points on the 2D area array camera comprises: The X-axis of the line structured light vision model is the horizontal axis of the photosensitive surface, and the Y-axis is the vertical axis of the photosensitive surface; The internal parameters of the 2D array camera are obtained as follows: ; Then the relationship between the image coordinates of a light strip point on the 2D array camera and its 3D coordinates is: ; After the line structured light vision model is calibrated, the coordinates of the laser surface under the 2D array camera are: ; From the above equation, the 3D coordinates of a light strip point on the 2D array camera can be obtained as: ； Where k represents the distortion coefficient, Respectively represent the focal length in the x-axis and y-axis directions, in pixels; is the coordinate of the principal point of the image, that is, the position of the point where the optical axis intersects the imaging plane in the image coordinate system, usually located at the center of the image, corresponds to the position in the x-axis direction, and Corresponding to the position on the y-axis, x represents the horizontal position of the mark, y represents the vertical position of the target point, and z represents the distance between the target point and the camera. represents the position along the x-axis on the image plane, represents the position along the y-axis on the image plane, a is the component of the normal vector in the x-axis direction, b is the component of the normal vector in the y-axis direction, and c is the component of the normal vector in the z-axis direction. T Represents matrix transpose.

3. The joint calibration method of a 2D area array camera and a line laser 3D sensor according to claim 1, characterized in that: The cylindrical features are installed in parallel on a plane, any three of the central axes of the multiple cylinders are not coplanar, and the relative positional relationship between the cylinders remains unchanged after installation.

4. The method for joint calibration of a 2D area array camera and a line laser 3D sensor according to claim 1, characterized in that: The target is placed at least once to complete the calibration.

5. The joint calibration method of a 2D area array camera and a line laser 3D sensor according to claim 1, characterized in that: The target placement can obtain the intersection line between the laser plane and the three cylinders, and is within the acquisition range of the 3D sensor and the 2D array camera at the same time.

Images