A coordinate mapping method for depth camera and visible light camera based on Zhang Zhengyou calibration method

Abstract

The present invention discloses a method for mapping the coordinate systems of a depth camera and a visible light camera based on the Zhang Zhengyou calibration method. When a three-dimensional model of a bolt in the depth camera coordinate system has been obtained, four points are first selected around the three-dimensional model of the bolt; a calibration plate is prepared, and the calibration plate is photographed with a depth camera and a visible light camera to obtain a depth map and a visible light map of the calibration image; the calibration image is processed using the Zhang Zhengyou calibration method, images with large errors are removed, and distortion parameters are selected to correct the distortion of the calibration results to obtain a camera intrinsic parameter matrix and an inter-camera extrinsic parameter matrix; the point coordinates are converted from the depth camera coordinate system to the visible light camera coordinate system using the camera extrinsic parameter matrix, and the points are mapped to the image pixel coordinate system using the visible light camera intrinsic parameter matrix. Finally, the minimum circumscribed rectangle of the quadrilateral formed by the four points is taken to extract the target mask. This method can effectively avoid background interference when directly extracting the target bolt mask from the visible light image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision 3D point cloud data processing algorithms, and in particular to a coordinate system mapping method for a depth camera and a visible light camera based on the Zhang Zhengyou calibration method. Background Art

[0002] Tall railway bridge piers are distributed in special environments such as water areas or valleys. In order to detect the bolt distribution, height and other information on the pillar foundation interface, on-site implementation personnel need to set up climbing tools, climb to the top of the tall bridge piers to carry out operations, and then remove the climbing tools after the operation is completed.

[0003] The main reasons why the inspection of the foundation interface of high bridge piers urgently needs to improve the working method are:

[0004] 1) The operation is difficult and dangerous. Workers are prone to danger when climbing or descending from high bridge piers, which poses serious safety hazards.

[0005] 2) The measurement cycle is long, and the entire preparation work is complex and difficult, requiring a lot of manpower and time, affecting the project construction progress.

[0006] 3) The data is not easy to save and analyze, and subsequent analysis and review are difficult.

[0007] Moreover, because the background colors of bolts and pillars under railway working conditions are highly similar, background interference has a significant impact on bolt segmentation in visible light images, and dimensional measurement is difficult in two-dimensional images. Therefore, a method that can effectively reduce background interference and clearly identify bolts needs to be developed. Summary of the Invention

[0008] The present invention provides a depth camera and visible light camera coordinate system mapping method based on Zhang Zhengyou calibration method, which can map the point coordinates in the bolt three-dimensional image to the visible light image to obtain an effective bolt mask.

[0009] In order to solve the above technical problems, the present invention adopts a depth camera and visible light camera coordinate system mapping method based on Zhang Zhengyou calibration method, which includes the following steps:

[0010] 1) Fix the depth camera and visible light camera, shoot the calibration plate to obtain the calibration image; calibrate the calibration image using Zhang Zhengyou calibration method, and remove the calibration images with large reprojection errors;

[0011] 2) Perform distortion correction on the calibration results of step 1), derive the intrinsic parameter matrix and inter-camera extrinsic parameter matrix of the depth camera and visible light camera, and transpose them, finally obtaining the conversion matrix from the visible light camera coordinate system to the image coordinate system and the conversion matrix from the depth camera coordinate system to the visible light camera coordinate system;

[0012] 3) Extract four points around the bolt 3D model in the depth camera coordinate system and obtain the coordinate values ​​of the four points;

[0013] 4) Mapping the coordinates of the four points obtained in step 3) from the depth camera coordinate system to the visible light camera coordinate system to obtain the coordinates of the four points in the visible light camera coordinate system;

[0014] 5) Map the coordinates of the four points in the visible light camera coordinate system to the visible light image of the bolt using the visible light camera intrinsic parameter matrix, convert from the visible light camera coordinate system to the image coordinate system, and finally convert to the pixel coordinate system to obtain the pixel coordinates of the four points;

[0015] 6) The minimum circumscribed rectangle of the quadrilateral is formed based on the pixel coordinates of the four points and the image is intercepted to obtain the bolt mask, completing the mapping of the bolt in the coordinate system.

[0016] As a preferred embodiment of the present invention, step 1) is specifically as follows: fix the depth camera and visible light camera on their respective gimbals and keep their relative positions fixed, prepare a calibration plate with a black and white grid side length of 50-80 mm, and use the depth camera and visible light camera to simultaneously shoot 30-60 groups of calibration pictures in different postures; convert the calibration pictures taken by the depth camera and visible light camera to the same size and calibrate them using the Zhang Zhengyou calibration method, and eliminate calibration pictures with large errors.

[0017] As a preferred solution of the present invention, the distortion correction in step 2) is: selecting distortion correction parameters according to the calibration result of step 1) to perform distortion correction.

[0018] As a preferred embodiment of the present invention, step 3) is specifically as follows: extracting the median X coordinate of all points in the bolt 3D model in the depth camera coordinate system, and then adding or subtracting the bolt radius value r from the median X coordinate; extracting the maximum and minimum Y coordinates of all points in the bolt 3D model; extracting the average Z coordinate of all points in the bolt 3D model, thereby obtaining the coordinate values ​​of four points around the bolt.

[0019] As a preferred solution of the present invention, step 4) is specifically as follows: the coordinates of the four points obtained in step 3) are corrected into homogeneous coordinate vectors, and the coordinate homogeneous vectors in the visible light camera coordinate system are obtained by converting the external parameter matrix between the depth camera and the visible light camera.

[0020] As a preferred embodiment of the present invention, step 5) is specifically as follows: mapping the coordinate homogeneous vectors of the four coordinate points through the visible light camera intrinsic parameter matrix to obtain the coordinate homogeneous vector in the camera image coordinate system, correcting the Z coordinate value to obtain the coordinate homogeneous vector in the camera pixel coordinate system, and extracting the pixel coordinates of the points around the bolt in the visible light image from the coordinate vector.

[0021] As a preferred solution of the present invention, step 6) is specifically as follows: analyzing the pixel coordinates of the four points, taking the maximum and minimum values ​​of the horizontal and vertical coordinates respectively, and expanding the pixel values ​​outward to obtain the circumscribed rectangle of the four points, intercepting the rectangular image to obtain the bolt mask, and completing the mapping of the bolt in the coordinate system.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] In the depth camera, the bolts have geometric information that is obviously different from the background. The three-dimensional model of the bolts is obtained by using the existing three-dimensional image target detection and segmentation algorithm. After taking points around the model, the mapping algorithm of the present invention is used to map the points to the visible light image, intercept the bolt mask, and then perform visible light image algorithm analysis, which can effectively reduce background interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the method of the present invention;

[0025] Figure 2 This is a three-dimensional model diagram of a single bolt;

[0026] Figure 3 This is a visible light image of the bolt support;

[0027] Figure 4 This is the visible light camera calibration map;

[0028] Figure 5 It is the depth camera calibration map;

[0029] Figure 6 This is the calibration diagram of the MATLAB Zhang Zhengyou calibration method app;

[0030] Figure 7 Take the point graph for the X coordinate;

[0031] Figure 8 Mapping three-dimensional coordinate points to visible light images;

[0032] Figure 9 Bolt mask diagram. DETAILED DESCRIPTION

[0033] The present invention will be further described and illustrated below in conjunction with specific embodiments. The embodiments are merely illustrative of the present disclosure and do not limit its scope. The technical features of the various embodiments of the present invention may be combined accordingly, provided that there is no conflict between them.

[0034] like Figure 2 As shown in , it is a 3D model of the bolt, which is obtained by using the existing 3D image target detection and segmentation algorithm; Figure 3As shown in the figure, it is an RGB image of a pillar interface with bolts. The background is complex and it is difficult to segment the foreground and background. In order to reduce background interference, Figure 1 As shown, the present invention provides a method for mapping the coordinate systems of a depth camera and a visible light camera based on the Zhang Zhengyou calibration method. The specific steps are as follows:

[0035] 1) Fix the depth camera and visible light camera, shoot the calibration plate to obtain the calibration image; calibrate the calibration image using Zhang Zhengyou calibration method, and remove the calibration images with large reprojection errors to ensure that the reprojection error is within one pixel;

[0036] 2) Perform distortion correction on the calibration results of step 4), derive the intrinsic parameter matrix and inter-camera extrinsic parameter matrix of the depth camera and visible light camera, and transpose them, finally obtaining the conversion matrix from the visible light camera coordinate system to the image coordinate system and the conversion matrix from the depth camera coordinate system to the visible light camera coordinate system;

[0037] 3) Extract four points around the bolt 3D model in the depth camera coordinate system and obtain the coordinate values ​​of the four points;

[0038] 4) Mapping the coordinates of the four points obtained in step 3) from the depth camera coordinate system to the visible light camera coordinate system to obtain the coordinates of the four points in the visible light camera coordinate system;

[0039] 5) Map the coordinates of the four points in the visible light camera coordinate system to the visible light image of the bolt using the visible light camera intrinsic parameter matrix, convert from the visible light camera coordinate system to the image coordinate system, and finally convert to the pixel coordinate system to obtain the pixel coordinates of the four points;

[0040] 6) According to the pixel coordinates of the four points, form the minimum circumscribed rectangle of the quadrilateral and intercept the image to obtain the bolt mask.

[0041] In a specific embodiment of the present invention, first fix the depth camera and the visible light camera, prepare a black and white grid calibration plate with a grid side length of 50mm, and use two cameras to simultaneously shoot calibration pictures of different postures, such as Figure 4 As shown in , the visible light camera captures the visible light image; Figure 5 As shown in FIG, the depth camera captures a depth map; there are 55 groups of visible light maps and depth maps.

[0042] Use the Stereo Camera Calibrator app that comes with MATLAB for camera calibration. The present invention uses MATLAB2020a. First, convert the depth image and visible light image to the same size. The image size used in the present invention is 640X480. Open the Stereo Camera Calibrator app for calibration, eliminate the images with large errors to ensure that the reprojection error is within 1 pixel, and select the distortion correction parameters according to the calibration results for distortion correction, such as Figure 6 Finally, the visible light camera internal parameter rotation matrix R is derived rgb , translation matrix T rgb and the camera external parameter rotation matrix R d2r , translation matrix T d2r , transpose these four matrices to obtain matrices Finally, the conversion matrix T from the visible light camera coordinate system to the image coordinate system is obtained rgb2img , the transformation matrix T from the depth camera coordinate system to the visible light camera coordinate system depth2rgb .

[0043] in:

[0044]

[0045]

[0046] The 3D model of the bolt is obtained by using the existing 3D image target detection and segmentation algorithm; Figure 7 As shown, take the median X coordinate of all points in the bolt 3D model. med , then increase or decrease the bolt radius length r along the X coordinate axis to obtain X med +r、X med -r, take the maximum and minimum Y coordinate values ​​of the bolt max 、Y min , take the average Z coordinate value Z med , get the four homogeneous coordinates X1-X4 in the depth camera coordinate system and the coordinate vector composed of four coordinate points as follows:

[0047] X1=(X med -rY max Z med 1)

[0048] X2=(X med -rY min Z med 1)

[0049] X3=(X med +rY max Z med1)

[0050] X4=(X med +rY min Z med 1)

[0051]

[0052] Will Convert to the visible light camera coordinate system to get the coordinates of the bolt in the visible light camera coordinate system Then convert from the visible light camera coordinate system to the image coordinate system, and finally to the pixel coordinate system to obtain the pixel coordinates Right now:

[0053]

[0054]

[0055]

[0056] i=1,2,3;j=1,2,3,4

[0057] Get the coordinate vector in the pixel coordinate system Then, analyze the maximum and minimum values ​​of the four coordinate points and take the minimum circumscribed rectangle of the four points to get the following: Figure 8 The bolt frame shown, the bolt mask is cut as Figure 9 As shown, the bolt has been cut out of the visible light image, and the background interference is greatly reduced.

[0058] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. Persons skilled in the art will readily appreciate that variations and modifications may be made without departing from the scope of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A method for mapping the coordinate systems of a depth camera and a visible light camera based on Zhang Zhengyou's calibration method, characterized in that: The following steps are involved: 1) Fix the depth camera and visible light camera and shoot the calibration plate to obtain calibration images; calibrate the calibration images using Zhang Zhengyou's calibration method, and remove calibration images with large reprojection errors; 2) Perform distortion correction on the calibration results of step 1), derive the intrinsic parameter matrix of the depth camera and visible light camera and the inter-camera extrinsic parameter matrix and transpose them, and finally obtain the conversion matrix from the visible light camera coordinate system to the image coordinate system and the conversion matrix from the depth camera coordinate system to the visible light camera coordinate system; 3) Extract four points around the bolt 3D model in the depth camera coordinate system and obtain the coordinate values ​​of the four points; Step 3) specifically comprises: extracting the median X coordinate of all points in the bolt 3D model in the depth camera coordinate system, and then adding or subtracting the bolt radius r from the median X coordinate; extracting the maximum and minimum Y coordinates of all points in the bolt 3D model; and extracting the average Z coordinate of all points in the bolt 3D model, thereby obtaining the coordinate values ​​of four points around the bolt; 4) Map the coordinates of the four points obtained in step 3) from the depth camera coordinate system to the visible light camera coordinate system to obtain the coordinates of the four points in the visible light camera coordinate system; The step 4) specifically comprises: correcting the coordinates of the four points obtained in step 3) into homogeneous coordinate vectors, and converting the external parameter matrix between the depth camera and the visible light camera to obtain the coordinate homogeneous vector in the visible light camera coordinate system; 5) Map the coordinates of the four points in the visible light camera coordinate system to the visible light image of the bolt using the visible light camera intrinsic parameter matrix, convert from the visible light camera coordinate system to the image coordinate system, and finally convert to the pixel coordinate system to obtain the pixel coordinates of the four points; Step 5) specifically includes: mapping the coordinate homogeneous vectors of the four coordinate points through the visible light camera intrinsic parameter matrix to obtain the coordinate homogeneous vector in the camera image coordinate system, correcting the Z coordinate value to obtain the coordinate homogeneous vector in the camera pixel coordinate system, and extracting the pixel coordinates of the points around the bolt in the visible light image from the coordinate vector; 6) The minimum circumscribed rectangle of the quadrilateral is formed based on the pixel coordinates of the four points and the image is intercepted to obtain the bolt mask, completing the mapping of the bolt in the coordinate system; The step 6) specifically includes: analyzing the pixel coordinates of the four points, taking the maximum and minimum values ​​of the horizontal and vertical coordinates respectively, and expanding the pixel values ​​outward to obtain the circumscribed rectangle of the four points, intercepting the rectangular image to obtain the bolt mask, and completing the mapping of the bolt in the coordinate system.

2. The method for mapping the coordinate systems of a depth camera and a visible light camera based on the Zhang Zhengyou calibration method according to claim 1, characterized in that: Step 1) specifically includes: fixing the depth camera and visible light camera on their respective gimbals and maintaining a fixed relative position; preparing a calibration plate with a black and white grid of 50-80 mm sides; and simultaneously capturing 30-60 sets of calibration images in different poses with the depth camera and visible light camera; converting the calibration images captured by the depth camera and visible light camera to the same size and performing calibration using the Zhang Zhengyou calibration method, eliminating calibration images with large errors.

3. The method for mapping the coordinate systems of a depth camera and a visible light camera based on the Zhang Zhengyou calibration method according to claim 1, wherein: The distortion correction described in step 2) is: selecting distortion correction parameters according to the calibration result of step 1) to perform distortion correction.

Images