Large-view multi-depth camera point cloud calibration splicing method

Abstract

The invention relates to a large-view multi-depth camera point cloud calibration splicing method, which belongs to the field of computer vision and specifically comprises the following steps of: (1) deploying a global vision system; (2) synchronously exposing the camera; (3) calculating external parameters of the camera; (4) unifying a global coordinate system; and (5) splicing point cloud integrated depth images. According to the invention, the whole belt can be covered with as few cameras as possible in a detection scene of the conveying belt with the same length, so that the cost can be saved; unification of multi-camera coordinate systems is completed through a multi-camera and few-common-view global calibration technology, and a new multi-sensor calibration method is provided.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a method for calibrating and stitching point clouds of a large-viewing-field multi-depth camera. Background Art

[0002] During the process of transporting coal by belt in mines, the data of the entire belt needs to be monitored and analyzed. A single depth camera cannot cover the entire belt scene. Multiple cameras need to be deployed to cover the entire belt scene. The common field of view between multiple cameras is small. A new calibration scheme is needed to unify the coordinate system of multiple depth cameras and realize the stitching of the entire belt scene.

[0003] Plane calibration schemes are usually used to calibrate the intrinsic parameters of the camera and the extrinsic parameters of the camera of a specified plane. In a plane production line where the camera is fixed, a plane calibration method is usually used to calibrate the relationship between the camera and the plane. This scheme mainly calibrates the rotational transformation relationship between the coordinate system of the camera and the production line plane through the constraints of a fixed plane. However, previous application scenarios were mostly for a single camera.

[0004] The existing binocular camera calibration solution requires a certain common field of view between the two cameras. The problem is that too large a common field of view will lead to more cameras being deployed for the same belt length, which makes the overall solution costly. Moreover, pairwise calibration will lead to the transmission of calibration errors between multiple cameras, resulting in poor final stitching results.

[0005] The unified calibration method based on multiple cameras is also widely used in the field of autonomous driving, but it is usually a stereo calibration, which is achieved by establishing the 3D coordinates of the calibration room and the calibration points. It is necessary to use more precise instruments such as total stations to measure the 3D position information of the calibration points to achieve calibration between cameras. The calibration process is complicated and cumbersome, which is not conducive to on-site engineering deployment and implementation. Summary of the invention

[0006] This invention expands the single-camera plane calibration method and proposes a large-view multi-depth camera point cloud calibration and splicing method. A large-area plane calibration plate is used to calibrate the deployed multi-depth cameras at the same time. The calibration method is simple to operate and highly efficient. It can be applied to the full-view splicing of coal mine belt transportation scenes. This invention aims to propose a mine hoist wire rope defect detection method based on feature embedding to improve the accuracy and performance of wire rope defect detection. The specific detection method flow chart is as follows:

[0007] The present invention proposes a method for calibrating and stitching point clouds of large-viewing-field multi-depth cameras, comprising the following steps:

[0008] (1) Global vision system deployment: Based on the length of the test belt, multiple depth cameras are used to cover the entire belt.

[0009] (2) Camera synchronous exposure: The cameras are synchronously exposed through hard triggering. Multiple cameras simultaneously shoot the calibration cloth and find the calibration corner points through image processing algorithms.

[0010] (3) Calculate the external parameters of the camera: According to the principle of pinhole imaging, the rotation matrix and translation matrix of multiple cameras are obtained [R i , t i ];

[0011] (4) Unified global coordinate system: After the external parameters of multiple cameras are calibrated, the depth camera obtains the point cloud coordinate information and transforms R through the calibrated rotation and translation base，i , T base，i Relationship unified coordinate system, where i represents the coordinate system of the i-th camera, and the origin of the coordinate system is the origin of the belt calibration cloth;

[0012] (5) Point cloud integration and depth image stitching: Get the point cloud coordinates of the actual depth camera {X i , Y i , Z i}, transform the coordinate system through rotation and translation relationship,

[0013] Among them, in step (2), the image processing algorithm adopts Harris corner detection algorithm, uses a fixed window (takes a neighborhood window of a certain pixel) to slide in any direction on the image, and compares the degree of change of pixel grayscale in the window before and after sliding. If there is a large grayscale change when sliding in any direction, it can be considered that there is a corner point in the window.

[0014] Step (3) specifically includes: firstly, using the Zhang Zhengyou plane calibration algorithm to obtain the camera's intrinsic parameter matrix K, and remove the image distortion, and obtain the pixel value of the image after the distortion is removed as {u, u}, and then calculate the pixel value of the plane pixel point {u i , v i}corresponding to the global coordinate point {X i , Y i , Z i}, since the world global coordinate system is a plane coordinate system, z can be set to 0, and the global coordinate system can be degenerated to {X i , Y i}, the upper left corner of the calibration plate is the origin of the coordinate system, {X0, Y0} = {0, 0}, 40 cm is a grid, and the actual position coordinates of the plane point are {X i , Y i}={40*i, 40*i}, find the corresponding relationship, and then use the least squares method to obtain the homogeneous transformation matrix The world coordinate system coincides with the plane coordinate system, z = 0, then the above formula is simplified to H=K[RT] / s=K[r1 r2 t] / s. According to the properties of the matrix, as long as the coordinates of the four pairs of points are known, the rotation matrix R and the translation matrix t can be obtained. The rotation matrix and translation matrix [R i , t i ].

[0015] The beneficial effects of the present invention are as follows: the present invention can cover the entire belt with as few cameras as possible in the detection scene of the transport belt of the same length, which can save costs; the global calibration technology of multiple cameras with a few common fields of view is used to unify the coordinate systems of multiple cameras, providing a new method for multi-sensor calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the flow of the method for calibrating and stitching point clouds of a large field of view and multi-depth camera of the present invention;

[0017] Figure 2 A deployment design diagram for the global vision system of the present invention;

[0018] Figure 3 The original design drawing of the calibration cloth of the present invention;

[0019] Figure 4 The actual image of the calibration cloth taken by the camera 1 of the present invention;

[0020] Figure 5 This is the actual image of the calibration cloth taken by the camera 2 of the present invention. DETAILED DESCRIPTION

[0021] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0022] like Figure 1 As shown, the present invention proposes a method for calibrating and stitching point clouds of large-field-of-view multi-depth cameras, which specifically includes: (1) global vision system deployment; (2) camera synchronous exposure; (3) calculation of camera external parameters; (4) unified global coordinate system; (5) point cloud integration and depth image stitching.

[0023] Specifically, in step (1), the deployment design of the global vision system deployment is as follows: Figure 2 As shown, according to the current experimental belt length and camera field of view, the panoramic belt is covered by 3 depth cameras;

[0024] Specifically, in step (2), the cameras are synchronously exposed through hard triggering, and the three cameras simultaneously shoot the calibration cloth, such as Figure 3 As shown, the calibrated corner points are found through the Harris corner detection algorithm;

[0025] Specifically, in step (3): according to the principle of pinhole imaging of the camera, the intrinsic parameter matrix K of the camera is first obtained by Zhang Zhengyou plane calibration algorithm, and the image distortion is removed to obtain the pixel value of the image after the distortion is removed as {u, v}. i , v i}corresponding to the global coordinate point {X i , Y i , Z i}, since the world global coordinate system is a plane coordinate system, we can set Z = 0, and the global coordinate system can be degenerated into {X i , Y i}, the upper left corner of the calibration plate is the origin of the coordinate system, {X0, Y0} = {0, 0}, 40 cm is a grid, and the actual position coordinates of the plane point are {X i , Y i}={40*i, 40*i}, find the corresponding relationship, and then use the least squares method to obtain the homogeneous transformation matrix The world coordinate system coincides with the plane coordinate system, z = 0, then the above formula is simplified to H = K[R t] / s = K[r1 r2 t] / s. According to some properties of the matrix, as long as the coordinates of the four pairs of points are known, the rotation matrix R and the translation matrix t can be obtained. The rotation matrix and translation matrix [R i , t i ].

[0026] Specifically, in step (4), after the three camera external parameters are calibrated, the depth camera obtains the point cloud coordinate information and transforms R base，i , T base，i Relationship unified coordinate system, where i represents the coordinate system of the i-th camera, and the origin of the coordinate system is the origin of the belt calibration cloth;

[0027] Specifically, in step (5), point cloud integration depth image stitching: Get the point cloud coordinates of the actual depth camera {X i , Y i , Z i}, transform the coordinate system through rotation and translation relationship,

[0028] The following example is used to explain this in detail.

[0029] like Figure 4 , Figure 5 As shown in the figure, the stitching process of the images taken by two cameras is as follows. First, the calibration cloth is laid flat on the conveyor belt, and the two cameras are triggered to shoot. The images taken are dedistorted to obtain the following images. Figure 4) as an example, the coordinates of the four corner points actually marked in the calibration coordinate system are (280, 40, 0), (440, 40, 0), (280, 120, 0), (440, 120, 0), and the corresponding extracted pixel coordinates are as follows: (124, 113), (611, 111), (122, 357), (614, 356). The intrinsic parameters of camera 1 are obtained through factory calibration, and the intrinsic parameter matrix is

[0030]

[0031] Therefore, the rotation transformation relationship matrix between camera 1 and the calibration cloth is obtained as follows:

[0032]

[0033] The translation matrix is,

[0034] T=[-3507.39860732,-782.97708473,1771.03559076]

[0035] Similarly, we can find the camera 2 ( Figure 5 )’s rotation and translation relationship is used to normalize the coordinate system to complete the calibration of multiple cameras and then realize the point cloud stitching process.

[0036] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for calibrating and stitching point clouds of large field of view multi-depth cameras, characterized in that: The steps include: (1) Global vision system deployment: Based on the length of the test belt, multiple depth cameras are used to cover the entire belt. (2) Camera synchronous exposure: The cameras are synchronously exposed through hard triggering. Multiple cameras simultaneously shoot the calibration cloth and find the calibration corner points through image processing algorithms. (3) Calculate the external parameters of the camera: According to the principle of pinhole imaging, the rotation matrix and translation matrix of multiple cameras are obtained [R i , t i ]; (4) Unified global coordinate system: After the external parameters of multiple cameras are calibrated, the depth camera obtains the point cloud coordinate information and transforms R through the calibrated rotation and translation base，i , T base，i Relationship unified coordinate system, where i represents the coordinate system of the i-th camera, and the origin of the coordinate system is the origin of the belt calibration cloth; (5) Point cloud integration and depth image stitching: Get the point cloud coordinates of the actual depth camera {X i , Y i , Z i }, transform the coordinate system through rotation and translation relationship, 2. The method for calibrating and stitching point clouds of a large field of view multi-depth camera according to claim 1, characterized in that: In step (2), the image processing algorithm adopts the Harris corner detection algorithm, which uses a fixed window to slide in any direction on the image, and compares the degree of change of the pixel grayscale in the window before and after the sliding. If there is sliding in any direction, which will produce a large grayscale change, it is considered that there is a corner point in the window.

3. The method for calibrating and stitching point clouds of a large field of view multi-depth camera according to claim 2, characterized in that: Step (3) specifically includes: firstly, using Zhang Zhengyou plane calibration algorithm to obtain the camera's intrinsic parameter matrix K, and remove the image distortion, obtain the pixel value of the image after the distortion is removed {u, v}, and calculate the pixel value of the plane pixel point {u i , v i }corresponding to the global coordinate point {X i , Y i , Z i }, since the world global coordinate system is a plane coordinate system, set z = 0, the global coordinate can be degenerated into {X i , Y i }, the upper left corner of the calibration plate is the origin of the coordinate system, {X0, Y0} = {0, 0}, 40 cm is a grid, and the actual position coordinates of the plane point are {X i , Y i }={40*i, 40*i}, find the corresponding relationship, and obtain the homogeneous transformation matrix through the least squares method The world coordinate system coincides with the plane coordinate system, z = 0, then the above formula is simplified to H=K[Rt ] / s=K[r1 r2 t] / s, according to the properties of the matrix, the rotation matrix and translation matrix [R i , t i ].

Images