A method for calibrating a radar camera without a calibration board in a dynamic vehicle cabin environment

Abstract

The present invention discloses a calibration method for radar-camera without calibration board in a dynamic carriage environment, which relates to the technical field of calibration methods for radar-camera without calibration board, and includes the following steps: S1, initial parameter estimation, estimating the movements of the radar and the camera relative to the carriage respectively from the forward and backward movements of the carriage; S2, external parameter estimation of the radar-camera, solving the external parameters of the radar-camera according to the results of the initial parameter estimation in S1; S3, parameter optimization, optimizing the external parameters of the radar-camera in S2 to improve the accuracy. The method of the present invention does not require the use of a specific calibration board for collecting point cloud and image data, and at the same time eliminates the need for on-site manual intervention, and can remotely calculate the required radar-camera external parameter. In the production operation site, the multi-sensor fusion of lidar and camera plays a key role in the train carriage partial load monitoring system, and the prerequisite for the joint application of lidar and camera is to perform joint calibration to calculate the external parameters of lidar and camera.

Description

Technical Field

[0001] The present invention relates to the technical field of radar-camera calibration methods without calibration plates, and particularly to a radar-camera calibration method without calibration plates in a dynamic carriage environment. Background Art

[0002] In past research work, many scientific researchers have proposed theoretical methods and calibration tools for the joint calibration of lidar and cameras, and many of them have achieved relatively good results. However, in actual production operations, when a train loads goods, it is transported through a chute on the loading building, so the entire equipment will generate relatively large vibrations, which will in turn drive the entire loading building to vibrate, resulting in the relevant positions of the lidar and camera installed on it shifting. Therefore, the parameters of the joint calibration of the lidar and camera need to be calibrated frequently. Traditional lidar and camera calibration methods have some disadvantages, such as time-consuming, low accuracy, and the need for on-site calibration by personnel. Summary of the Invention

[0003] The purpose of the present invention is to solve the above technical problems, and a radar-camera calibration method without calibration plates in a dynamic carriage environment is proposed.

[0004] To achieve the above purpose, the present invention adopts the following technical solutions:

[0005] A radar-camera calibration method without calibration plates in a dynamic carriage environment includes the following steps:

[0006] S1, Initial parameter estimation, estimating the motions of the radar and the camera relative to the carriage separately from the forward and backward motions of the carriage;

[0007] S2, Estimation of the external parameters of the radar-camera, solving the external parameters of the radar-camera according to the results of the initial parameter estimation in S1;

[0008] S3, Parameter optimization, optimizing the external parameters of the radar-camera in S2 to improve the accuracy.

[0009] Preferably, the initial parameter estimation includes radar transformation parameter estimation and camera transformation parameter estimation.

[0010] Preferably, the radar transformation parameter estimation includes the following steps:

[0011] A1, First, perform radar coordinate system transformation, manually select four corner points in the carriage point cloud to obtain the coordinates of each point;

[0012] A2, Taking the carriage corner points as the coordinate origin and the carriage border as the coordinate axis direction, construct a carriage coordinate system;

[0013] A3, According to the carriage specifications, the coordinates of each corner point in the carriage coordinate system can be determined, and the centroid of each point is calculated;

[0014] A4. Through formula calculation, the transformation from the radar coordinate system to the carriage coordinate system is achieved.

[0015] A5. Segment the complete point cloud of a single carriage, use the ICP algorithm to register the carriage point cloud, and solve the pose transformation of the radar relative to the carriage.

[0016] Preferably, the estimation of the camera transformation parameters includes the following steps:

[0017] B1. Take the upper left corner point of the carriage in the actual environment as the origin of the world coordinate system;

[0018] B2. Perpendicular to the plane formed by the carriage corner points;

[0019] B3. Establish coordinates according to the known length and width of the carriage;

[0020] B4. Calculate the pose transformation of the camera within the time through cosine theorem constraints, the coordinates of the four carriage corner points in the image, and coplanarity constraints of the corner points;

[0021] B5. Combine the known carriage specifications to obtain the pose transformation of the camera with a true scale within the time.

[0022] Preferably, the ways of establishing coordinates in B3 are pixel coordinate system, image coordinate system, and camera coordinate system.

[0023] Preferably, the estimation of the external parameters of the radar and camera includes the following steps:

[0024] C1. Establish the rotation matrix and translation matrix of the camera relative to the carriage;

[0025] C2. Establish the rotation matrix and translation matrix of the radar relative to the carriage;

[0026] C3. Solve the external parameter matrix of the radar and the camera;

[0027] Preferably, the parameter optimization includes the optimization of the camera pose change parameters and the optimization of the radar and camera parameters.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] To sum up, the method of the present invention does not need to use a specific calibration board to collect point cloud and image data, and at the same time eliminates manual on-site intervention, and can remotely calculate the required external parameters of the radar-camera. In the production operation site, the multi-sensor fusion of lidar and camera plays a key role in the offloading monitoring system of train carriages, and the prerequisite for the joint application of lidar and camera is to perform joint calibration to calculate the external parameters of lidar and camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is the method flow chart of the present invention;

[0031] Figure 2 is the diagram for manually selecting the carriage corner points in the point cloud of the present invention;

[0032] Figure 3 is the schematic diagram of the transformation between the radar coordinate system and the carriage coordinate system of the present invention;

[0033] Figure 4 is the schematic diagram of segmenting the train carriage from the original point cloud of the present invention;

[0034] Figure 5 is the two-frame point cloud diagrams in the same coordinate system of the present invention;

[0035] Figure 6 is the point cloud result diagram after ICP registration of the present invention;

[0036] Figure 7 is the camera imaging model of the present invention;

[0037] Figure 8 is the schematic diagram of the cosine theorem constraint. Detailed implementation manners

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments.

[0039] Refer to Figures 1-8 , a radar-camera calibration method without a calibration board in a dynamic carriage environment, including the following steps:

[0040] S1. Initial parameter estimation, estimating the movements of the radar and the camera relative to the carriage respectively from the forward and backward movements of the carriage;

[0041] S2. Radar-camera extrinsic parameter estimation, solving the extrinsic parameters of the radar-camera according to the results of the initial parameter estimation in S1;

[0042] S3. Parameter optimization, optimizing the extrinsic parameters of the radar-camera in S2 to improve the accuracy.

[0043] In the present invention, the initial parameter estimation includes radar transformation parameter estimation and camera transformation parameter estimation.

[0044] In the present invention, the radar transformation parameter estimation includes the following steps:

[0045] A1. First, perform radar coordinate system transformation, manually select four corner points in the carriage point cloud to obtain the coordinates of each point;

[0046] A2. Taking the corner points of the carriage as the coordinate origin and the carriage frame as the coordinate axis directions, a carriage coordinate system is constructed;

[0047] A3. According to the carriage specifications, the coordinates of each corner point in the carriage coordinate system can be determined, and the centroid of each point is calculated;

[0048] A4. Through formula calculation, the transformation from the radar coordinate system to the carriage coordinate system is realized.

[0049] In the present invention, the camera transformation parameter estimation includes the following steps:

[0050] B1. Taking the upper left corner point of the carriage in the actual environment as the origin of the world coordinate system;

[0051] B2. Perpendicular to the plane formed by the carriage corner points;

[0052] B3. Measuring the length and width of the carriage and establishing coordinates;

[0053] B4. Through formula calculation, the pose transformation of the camera within the time is obtained.

[0054] In the present invention, the way of establishing coordinates in B3 includes a pixel coordinate system, an image coordinate system, and a camera coordinate system.

[0055] In the present invention, the external parameter estimation of the radar and camera includes the following steps:

[0056] C1. Establishing a rotation matrix of the camera relative to the carriage;

[0057] C2. Establishing a rotation matrix of the radar relative to the carriage;

[0058] C3. Establishing a translation matrix and calculating through formulas.

[0059] In the present invention, the parameter optimization includes the optimization of the camera pose change parameters and the optimization of the radar and camera parameters.

[0060] A further detailed explanation of the present invention:

[0061] Radar transformation parameter estimation

[0062] First, the radar coordinate system transformation is performed. As shown in 2 the figure, four corner points are manually selected from the carriage point cloud to obtain the coordinates of each point;

[0063] Taking the carriage corner points as the coordinate origin and the carriage frame as the coordinate axis directions, as shown in 3 the figure, a carriage coordinate system is constructed;

[0064] According to the carriage specifications, the coordinates of each corner point in the carriage coordinate system can be determined First, find and The centroid of each point, as shown in the formula

[0065]

[0066]

[0067] N is the number of corresponding point pairs in the two coordinate systems. There are a total of 4 point pairs, so N is 4.

[0068] Calculate the covariance matrix H according to the formula, and perform SVD singular value decomposition on H, as shown in the formula

[0069]

[0070]

[0071] Solve the rotation and translation transformation matrix from the radar coordinate system to the carriage coordinate system according to the results of SVD , as shown in the formula.

[0072]

[0073]

[0074] Through the above steps, the transformation from the radar coordinate system to the carriage coordinate system is realized.

[0075] The ICP algorithm realizes the matching by iteratively optimizing the spatial transformation between point clouds. Its basic idea is to find the nearest neighbor point pairs in the two point clouds, calculate the registration error and adjust the transformation matrix through an optimization method, and repeat the iteration until the convergence condition is met. Collect the point clouds within time T , corresponding to the moments .

[0076] Before performing ICP point cloud registration, first perform manual point cloud segmentation, as shown in Figure 4 ;

[0077] Remove the background information that affects ICP point cloud registration, such as objects like the ground and railings, and only retain the point cloud of a complete carriage.

[0078] Next, select the point clouds of two adjacent frames within time t , , as shown in Figure 5 ;

[0079] The light-colored point cloud is , and the dark-colored point cloud is . Put the two frames of point clouds into the same coordinate system. It can be seen from the boxed area in the figure that the two frames of point clouds at different times do not coincide, especially the difference in the vehicle frame area is more obvious. Use the ICP algorithm to perform point cloud registration, and the registration result is shown in Figure6 as shown

[0080] After that, the transformation parameters of the point cloud within time t are obtained :

[0081]

[0082] After the transformation from the radar to the carriage coordinate system, the rotation and translation transformation of the radar relative to the carriage within time T can be directly obtained .

[0083] Camera transformation parameter estimation

[0084] By annotating the carriage corner points in the image, the coordinates of the carriage corner points on the image can be obtained

[0085] Taking the upper left corner point of the carriage in the actual environment as the origin of the world coordinate system The z-axis is perpendicular to the plane formed by the carriage corner points The x-axis is parallel to , The y-axis is parallel to . The coordinates of O are (0, 0, 0). The true length and width of the carriage are known. Let the length and width be a and b respectively, then The coordinates of A are , The coordinates of B are . The coordinates of the corresponding point in the pixel coordinate system are .

[0086] Pixel coordinate system. The most common digital image is composed of W×H pixels. Taking the upper left corner of the image as the origin, a two-dimensional pixel coordinate system is constructed , and each pixel in the image will have a unique coordinate value on the pixel coordinate system

[0087] Image coordinate system. The image coordinate system is a system that establishes the relationship between image pixels and the actual physical situation. Generally, the image coordinate system takes the center of the image as the origin and establishes a plane rectangular coordinate system , and its two coordinate axes are respectively parallel to those of the pixel coordinate system. The transformation relationship between the pixel coordinate system and the image coordinate system is as shown in the formula

[0088]

[0089] Among them, , are the horizontal and vertical coordinates of the pixel coordinate system , are the horizontal and vertical coordinates of the image coordinate system represents the width of the pixel in the x direction Represents the width of the pixel in the y - direction.

[0090] Camera coordinate system. The camera coordinate system is a bridge for establishing the physical relationship between a two - dimensional image and a three - dimensional space. The camera imaging model is shown in Figure 7 as follows;

[0091] To obtain the position of the target in the image in the world coordinate system, it is first necessary to construct the camera coordinate system and convert the physical relationship of the two - dimensional image to the three - dimensional space. The origin of the camera coordinate system is the optical center of the camera. Its z - axis points from the optical center along the optical axis direction. Its x - axis, y - axis, and z - axis are perpendicular to each other. A point in the camera coordinate system can be represented by From the formula, the relationship between the pixel coordinate system and the camera coordinate system can be known.

[0092]

[0093] Let the camera internal parameters be The three - dimensional coordinates on the normalized plane can be obtained:

[0094]

[0095] As shown in Figure 8 Let the camera optical center be , and there are two points in the world coordinate system, and is to distance;

[0096] Next, the problem is transformed into solving the camera pose by PnP. According to the cosine theorem, the constraint condition can be obtained:

[0097]

[0098] The coordinate on the normalized plane is The coordinate on the normalized plane is , then the unit vector of the vector is , the unit vector of the vector is . From the formula, it can be obtained that , is

[0099]

[0100] According to the vectors and it can be obtained:

[0101]

[0102] According to the point The coordinates in the normalized plane can be obtained The distance to the camera optical center , from = The coordinates in the camera coordinate system can be obtained From the perspective projection model: ,

[0103]

[0104] Also, since the coordinates in the world coordinate system are known, the rotation and translation matrix can be obtained , and the pose transformation of the camera within time T can be obtained

[0105] Solution of the extrinsic parameters of the lidar and camera

[0106] Let the rotation matrix of the camera relative to the carriage be , and the translation matrix be . Let the rotation matrix of the lidar relative to the carriage be , and the translation matrix be :

[0107]

[0108] By transforming and solving the above equation, the relationship between the camera motion and the lidar motion can be obtained:

[0109]

[0110] Because , is known, and according to the above equation, R is obtained by using non - linear optimization:

[0111]

[0112] After optimizing , substitute it into the equation to solve for t

[0113] Parameter optimization

[0114] Optimization of the camera pose change parameters

[0115] As shown in the formula, the camera motion is optimized by minimizing the projection error using the matching points and the error metric of the polar plane angle:

[0116]

[0117] , is the corresponding matching point, j is the number of matching points. Since the corner points of the forward and backward motion of the carriage are used for matching here, the maximum value of j is 4

[0118] Optimization of Radar Camera Parameters

[0119] Let be a point in the radar coordinate system. According to the work in the previous part of this chapter, the coordinates in the pixel coordinate system of the camera can be calculated :

[0120]

[0121] represents the projection from the camera coordinate system to the pixel coordinate system. For specific reference, see the formula. Thus, the projection of the three-dimensional point after the movement of the carriage in the image coordinate system can be obtained.

[0122]

[0123] By using the reprojection error and non-linear optimization to optimize the objective function and further optimize the parameters, as shown in the formula:

[0124]

[0125] Innovative points of the present invention:

[0126] This method innovatively uses the carriage in the on-site environment as a reference object, does not rely on a calibration board to provide features, avoids manual on-site calibration, and remotely calculates the external parameters of the radar and the camera.

[0127] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered within the protection scope of the present invention.

Claims

1. A method for calibrating a radar camera without a calibration board in a dynamic vehicle environment, characterized in that It includes the following steps: S1. Initial parameter estimation: Estimate the movements of the radar and the camera relative to the carriage from the forward and backward movements of the carriage respectively. S2. External parameter estimation of the radar and camera: Solve the external parameters of the radar and camera according to the results of the initial parameter estimation in S1. S3. Parameter optimization: Optimize the external parameters of the radar and camera in S2 to improve the accuracy. The initial parameter estimation includes radar transformation parameter estimation and camera transformation parameter estimation. The radar transformation parameter estimation includes the following steps: A1. First, perform radar coordinate system transformation. Manually select four corner points in the carriage point cloud to obtain the coordinates of each point. A2. Take the carriage corner points as the coordinate origin and the carriage border as the coordinate axis directions to construct a carriage coordinate system. A3. According to the carriage specifications, the coordinates of each corner point in the carriage coordinate system can be determined, and the centroid of each point can be calculated. A4. Through formula calculation, the transformation from the radar coordinate system to the carriage coordinate system is realized. The camera transformation parameter estimation includes the following steps: B1. Take the upper left corner point of the carriage in the actual environment as the origin of the world coordinate system. B2. Be perpendicular to the plane formed by the carriage corner points. B3. Measure the length and width of the carriage and establish coordinates. B4. Obtain the pose transformation of the camera within the time through formula calculation. The way to establish coordinates in B3 includes pixel coordinate system, image coordinate system and camera coordinate system. The external parameter estimation of the radar and camera includes the following steps: C1. Establish a rotation matrix of the camera relative to the carriage. C2. Establish a rotation matrix of the radar relative to the carriage. C3. Establish a translation matrix and calculate through formula. The parameter optimization includes camera pose change parameter optimization and radar and camera parameter optimization.

Images