Radar camera calibration-board-free calibration method in dynamic compartment environment

Abstract

The invention discloses a radar camera calibration-plate-free calibration method in a dynamic compartment environment, and relates to the technical field of radar camera calibration-plate-free calibration methods, and the method comprises the following steps: S1, initial parameter estimation: respectively estimating the movement of a radar and a camera relative to a compartment from the forward and backward movement of the compartment; s2, estimating external parameters of the radar camera, and solving the external parameters of the radar camera according to an initial parameter estimation result in S1; and S3, parameter optimization: optimizing the external parameters of the radar camera in the S2 to improve the precision. According to the method, a specific calibration plate is not needed to collect point cloud and image data, meanwhile, manual field intervention is omitted, and the required radar-camera external parameters can be remotely calculated. In a production operation site, multi-sensor fusion of a laser radar and a camera plays a key role in a train carriage unbalance loading monitoring system, and the premise of combined application of the laser radar and the camera is to carry out combined calibration and calculation of external parameters of the laser radar and the camera.

Description

Technical Field

[0001] The present invention relates to the technical field of radar camera calibration method without calibration plate, and in particular to a radar camera calibration method without calibration plate in a dynamic vehicle cabin environment. Background Art

[0002] In past research, many researchers have proposed theoretical methods and calibration tools for joint calibration of LiDAR and cameras, many of which have achieved relatively good results. However, in actual production operations, cargo is loaded on trains through chutes on the loading floor, so the entire equipment will vibrate greatly, which in turn causes the entire loading floor to vibrate, causing the relative positions of the LiDAR and cameras installed on it to shift. Therefore, the parameters of the joint calibration of LiDAR and cameras must be calibrated frequently. Traditional LiDAR and camera calibration methods have some disadvantages, such as time-consuming, low accuracy, and the need for on-site calibration. Summary of the invention

[0003] The purpose of the present invention is to solve the above technical problems and to propose a calibration method for a radar camera without a calibration plate in a dynamic cabin environment.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: A calibration method for a radar camera without a calibration plate in a dynamic vehicle cabin environment comprises the following steps: S1, initial parameter estimation, estimates the motion of the radar and camera relative to the car from the forward and backward motion of the car respectively; S2, radar camera external parameter estimation, solves the external parameters of the radar camera according to the results of the initial parameter estimation in S1; S3, parameter optimization, optimizes the external parameters of the radar camera in S2 to improve accuracy.

[0005] Preferably, the initial parameter estimation includes radar transformation parameter estimation and camera transformation parameter estimation.

[0006] Preferably, the radar transformation parameter estimation comprises the following steps: A1, first perform radar coordinate system transformation, manually select four corner points in the car point cloud, and obtain the coordinates of each point; A2, taking the corner point of the carriage as the coordinate origin and the carriage frame as the coordinate axis direction, construct the carriage coordinate system; A3, according to the car specifications, we can determine the coordinates of each corner point in the car coordinate system and find the centroid of each point; A4, through formula calculation, the transformation from radar coordinate system to car coordinate system is realized.

[0007] A5, segment the complete single-section carriage point cloud, use the ICP algorithm to align the carriage point cloud, and solve the position transformation of the radar relative to the carriage.

[0008] Preferably, the camera transformation parameter estimation comprises the following steps: B1, take the upper left corner of the car in the actual environment as the origin of the world coordinate system; B2, perpendicular to the plane formed by the corner points of the carriage; B3, establish coordinates based on the known length and width of the carriage; B4, the camera’s position transformation in time is calculated through the cosine theorem constraint, the coordinates of the four car corners in the image, and the corner coplanarity constraint; B5, combined with the known car specifications, obtains the camera's pose transformation with real scale in time.

[0009] Preferably, the ways of establishing coordinates in S3 include pixel coordinate system, image coordinate system and camera coordinate system.

[0010] Preferably, the radar camera extrinsic parameter estimation comprises the following steps: C1, establish the rotation matrix and translation matrix of the camera relative to the car; C2, establish the rotation matrix and translation matrix of the radar relative to the car; C3, solve the external parameter matrix of radar and camera; Preferably, the parameter optimization includes camera posture change parameter optimization and radar camera parameter optimization.

[0011] Compared with the prior art, the present invention has the following beneficial effects: In summary, the method of the present invention does not require the use of a specific calibration plate for the collection of point cloud and image data, and eliminates manual on-site intervention, and can remotely calculate the required radar-camera extrinsic parameters. In the production operation site, the multi-sensor fusion of laser radar and camera plays a key role in the train carriage overload monitoring system, and the premise of the joint application of laser radar and camera is to perform joint calibration to calculate the extrinsic parameters of the laser radar and camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of the method of the present invention; Figure 2 A diagram for manually selecting car corner points in a point cloud according to the present invention; Figure 3 Schematic diagram of the transformation between the radar coordinate system and the vehicle compartment coordinate system of the present invention; Figure 4 A schematic diagram of segmenting a train carriage from an original point cloud according to the present invention; Figure 5 These are two point cloud images in the same coordinate system of the present invention; Figure 6 This is the point cloud result image after ICP registration of the present invention; Figure 7 The camera imaging model of the present invention; Figure 8 Schematic diagram of the cosine theorem constraint. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0014] Reference Figure 1-Figure 8 A calibration method for a radar camera without a calibration plate in a dynamic vehicle cabin environment comprises the following steps: S1, initial parameter estimation, estimates the motion of the radar and camera relative to the car from the forward and backward motion of the car respectively; S2, radar camera external parameter estimation, solves the external parameters of the radar camera according to the results of the initial parameter estimation in S1; S3, parameter optimization, optimizes the external parameters of the radar camera in S2 to improve accuracy.

[0015] In the present invention, the initial parameter estimation includes radar transformation parameter estimation and camera transformation parameter estimation.

[0016] In the present invention, the radar transformation parameter estimation comprises the following steps: A1, first perform radar coordinate system transformation, manually select four corner points in the car point cloud, and obtain the coordinates of each point; A2, taking the corner point of the carriage as the coordinate origin and the carriage frame as the coordinate axis direction, construct the carriage coordinate system; A3, according to the car specifications, we can determine the coordinates of each corner point in the car coordinate system and find the centroid of each point; A4, through formula calculation, the transformation from radar coordinate system to car coordinate system is realized.

[0017] In the present invention, the camera transformation parameter estimation comprises the following steps: B1, take the upper left corner of the car in the actual environment as the origin of the world coordinate system; B2, perpendicular to the plane formed by the corner points of the carriage; B3, measure the length and width of the carriage and establish coordinates; B4, the camera’s position transformation in time is calculated by formula.

[0018] In the present invention, the ways of establishing coordinates in S3 include pixel coordinate system, image coordinate system and camera coordinate system.

[0019] In the present invention, the radar camera extrinsic parameter estimation includes the following steps: C1, establish the rotation matrix of the camera relative to the car; C2, establish the rotation matrix of the radar relative to the car; C3, establish the translation matrix and calculate it through the formula.

[0020] In the present invention, the parameter optimization includes camera posture change parameter optimization and radar camera parameter optimization.

[0021] The present invention is further explained in detail: Radar transform parameter estimation; First, the radar coordinate system is transformed, such as Figure 2 As shown, four corner points are manually selected in the car point cloud. , get the coordinates of each point; The corner point of the carriage is taken as the coordinate origin, and the carriage frame is taken as the coordinate axis direction, such as Figure 3 As shown, construct the carriage coordinate system; According to the car specifications, the coordinates of each corner point in the car coordinate system can be determined First ask and The centroid of each point is as follows: N is the number of corresponding pairs of points in the two coordinate systems. There are 4 pairs of points in total, so N is 4.

[0022] According to the formula, the covariance matrix H is obtained, and the SVD singular value decomposition of H is performed, as shown in the formula: According to the results of SVD, the rotation and translation transformation matrix from the radar coordinate system to the car coordinate system is solved , such as the formula: After the above steps, the transformation from the radar coordinate system to the vehicle coordinate system is realized.

[0023] The ICP algorithm achieves matching by iteratively optimizing the spatial transformation between point clouds. The basic idea is to find the nearest neighbor point pair in two point clouds, calculate the registration error, and adjust the transformation matrix through optimization methods, and repeat the iteration until the convergence condition is met. , corresponding to the time .

[0024] Before performing ICP point cloud registration, manual point cloud segmentation is first performed, such as Figure 4 As shown; The background information that affects the ICP point cloud registration, such as the ground, railings and other objects, is removed, and only the point cloud of a complete carriage is retained.

[0025] Next, select the point clouds of two adjacent frames within time t , ,like Figure 5 As shown; Light-colored point cloud , the dark point cloud is , put the two frames of point cloud into the same coordinate system. From the box area in the figure, we can see that the two frames of point cloud before and after the moment do not overlap, especially the difference in the car frame area is more obvious. Use the ICP algorithm to align the point cloud, and the alignment result is as follows Figure 6 As shown; Then the transformation parameters of the point cloud at time t are obtained : After the transformation from the radar to the carriage coordinate system, the rotation and translation transformation of the radar relative to the carriage within time T can be directly obtained. .

[0026] Camera transformation parameter estimation; By marking the corner points of the carriage in the image, the coordinates of the corner points of the carriage on the image can be obtained; Take the upper left corner of the car in the actual environment as the origin of the world coordinate system. The axis is perpendicular to the plane formed by the corner points of the carriage. Axis parallel to , Axis parallel to . The coordinates are (0,0,0), the actual length and width of the carriage are known, and if the length and width are a and b respectively, then The coordinates are , The coordinates are . The coordinates of the winning pixel coordinate system are .

[0027] Pixel coordinate system. The most common digital image is composed of W×H pixels. The upper left corner of the image is used as the origin to construct a two-dimensional pixel coordinate system. , each pixel in the image will have a unique coordinate value in the pixel coordinate system.

[0028] Image coordinate system, the image coordinate system is a system that establishes the relationship between image pixels and actual physical objects. Generally, the image coordinate system takes the center of the image as the origin and establishes a plane rectangular coordinate system. , which are parallel to the two coordinate axes of the pixel coordinate system. The transformation relationship between the pixel coordinate system and the image coordinate system is shown in the formula: in, , is the horizontal and vertical coordinates of the pixel coordinate system, , is the horizontal and vertical coordinates of the image coordinate system, Indicates the width of the pixel in the x direction, Indicates the pixel width in the y direction.

[0029] Camera coordinate system,The camera coordinate system is a bridge to establish the physical relationship between the two-dimensional image and the three-dimensional space.,The camera imaging model is shown in Figure 7; In order to obtain the position of the target in the image in the world coordinate system, we must first construct the camera coordinate system to transform the physical relationship of the two-dimensional image into three-dimensional space. The origin of the camera coordinate system is the optical center of the camera, and its z axis points from the optical center to the optical axis. Its x-axis, y-axis, and z-axis are perpendicular to each other. A point in the camera coordinate system can be To express it, the relationship between the pixel coordinate system and the camera coordinate system can be seen from the formula: Assume the camera internal parameters are The three-dimensional coordinates on the normalized plane can be obtained: like Figure 8 As shown, let the camera optical center be , there are two points in the world coordinate system , and for arrive distance; Next, the problem is transformed into PnP to solve the camera pose. According to the cosine theorem, the constraints are: The coordinates on the normalized plane are The coordinates on the normalized plane are , then the vector The unit vector is ,vector The unit vector is . From the formula we can get , for According to the vector and You can get: According to the point In the normalized plane coordinates, we can get Distance to the camera's optical center ,Depend on = Can get Coordinates in the camera coordinate system , from the perspective projection model: Because the coordinates in the world coordinate system are known, the rotation and translation matrix can be obtained , and get the camera’s pose transformation in T time.

[0030] Radar camera external parameter solution; Assume the rotation matrix of the camera relative to the car is , the translation matrix is , let the rotation matrix of the radar relative to the car be , the translation matrix is : By transforming the above formula, we can get the relationship between camera motion and radar motion: because , It is known that according to the above formula, nonlinear optimization is used to solve R: right After optimization, substitute it into the equation and solve it to get t.

[0031] Parameter optimization; Optimization of camera pose change parameters; The formula optimizes the camera motion by minimizing the projection error using the matching points and the epipolar plane angle error metric: , is the corresponding matching point, j is the number of matching points. Because the corner points of the car moving forward and backward are used for matching, the maximum value of j is 4.

[0032] Radar camera parameter optimization; set up is a point in the radar coordinate system. Based on the previous work in this chapter, we can calculate the coordinates in the camera pixel coordinate system. :

[0033] It represents the projection from the camera coordinate system to the pixel coordinate system. The specific reference formula is: By using the reprojection error and nonlinear optimization, the objective function is optimized and the parameters are further optimized as shown in the formula: .

[0034] The innovative features of the present invention are as follows: This method innovatively uses the car body in the on-site environment as a reference object, does not rely on the calibration plate to provide features, avoids manual on-site calibration, and remotely calculates the external parameters of the radar and camera.

[0035] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A calibration method for a radar camera without a calibration plate in a dynamic vehicle environment, characterized in that: The following steps are involved: S1, initial parameter estimation, estimates the motion of the radar and camera relative to the car from the forward and backward motion of the car respectively; S2, radar camera external parameter estimation, solves the external parameters of the radar camera according to the results of the initial parameter estimation in S1; S3, parameter optimization, optimizes the external parameters of the radar camera in S2 to improve accuracy.

2. The method for calibrating a radar camera without a calibration plate in a dynamic vehicle cabin environment according to claim 1, characterized in that: The initial parameter estimation includes radar transformation parameter estimation and camera transformation parameter estimation.

3. The method for calibrating a radar camera without a calibration plate in a dynamic vehicle cabin environment according to claim 2, characterized in that: The radar transformation parameter estimation comprises the following steps: A1, first perform radar coordinate system transformation, manually select four corner points in the car point cloud, and obtain the coordinates of each point; A2, taking the corner point of the carriage as the coordinate origin and the carriage frame as the coordinate axis direction, construct the carriage coordinate system; A3, according to the car specifications, we can determine the coordinates of each corner point in the car coordinate system and find the centroid of each point; A4, through formula calculation, the transformation from radar coordinate system to car coordinate system is realized.

4. The method for calibrating a radar camera without a calibration plate in a dynamic vehicle cabin environment according to claim 2, characterized in that: The camera transformation parameter estimation comprises the following steps: B1, take the upper left corner of the car in the actual environment as the origin of the world coordinate system; B2, perpendicular to the plane formed by the corner points of the carriage; B3, measure the length and width of the carriage and establish coordinates; B4, the camera’s position transformation in time is calculated by formula.

5. The method for calibrating a radar camera without a calibration plate in a dynamic vehicle cabin environment according to claim 4, characterized in that: The ways of establishing coordinates in S3 include pixel coordinate system, image coordinate system and camera coordinate system.

6. The method for calibrating a radar camera without a calibration plate in a dynamic vehicle cabin environment according to claim 1, characterized in that: The radar camera extrinsic parameter estimation comprises the following steps: C1, establish the rotation matrix of the camera relative to the car; C2, establish the rotation matrix of the radar relative to the car; C3, establish the translation matrix and calculate it through the formula.

7. The method for calibrating a radar camera without a calibration plate in a dynamic vehicle cabin environment according to claim 1, characterized in that: The parameter optimization includes camera posture change parameter optimization and radar camera parameter optimization.

Images