A method and apparatus for joint calibration of camera and radar

Abstract

This application provides a joint calibration method and apparatus for a camera and radar, comprising: with both the camera's intrinsic parameters and the radar calibrated on the vehicle, determining the coordinates of the calibration board's center in the pixel coordinate system when the calibration board is in at least three positions, and determining the coordinates of the calibration board's center in the radar coordinate system based on the point cloud information of the calibration board's center and / or the point cloud information of the points surrounding the calibration board's center; establishing a transformation relationship between the radar coordinate system and the pixel coordinate system for each position based on the coordinates of the calibration board's center in both the radar coordinate system and the pixel coordinate system; and performing joint calibration of the camera and radar based on the transformation relationship for each position to obtain the camera's rotation matrix and translation matrix. The calibration process of this application does not require manual measurement of horizontal and vertical offset distances, improving calibration accuracy and efficiency, exhibiting strong generalization and scalability, and having no requirements regarding the distance between the camera and the calibration board.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a camera and radar joint calibration method and device. BACKGROUND

[0002] In the fields of mobile robots, autonomous driving, assisted driving, and environmental perception, a single sensor cannot meet the sensing needs of complex environments, and multi-sensor fusion algorithms have become the mainstream algorithm. The multi-sensor fusion algorithm improves the precision of the algorithm by solving the time synchronization and spatial synchronization between sensors, and the spatial synchronization mainly involves joint calibration between sensors.

[0003] Current camera and radar joint calibration methods mainly include automatic joint calibration and manual joint calibration: automatic joint calibration is based on existing calibration tool libraries, for example, automatic joint calibration based on a checkerboard is implemented through Autoware, and automatic joint calibration based on feature points is implemented through Apollo; the process of manual joint calibration is as follows: first, measure the distance between the radar sensor and the ground, place a marker at a specified distance from the vertical point of the radar sensor ground, and according to the horizontal and vertical distances, the coordinates of the marker in the radar coordinate system can be obtained, and the coordinates of the marker in the camera coordinate system are determined, and the determined coordinates are converted to the pixel coordinate system, the relationship between the 2D pixel coordinate system and the 3D radar coordinate system is established, and the relationship formula for establishing the relationship between the two coordinate systems is solved by moving the marker.

[0004] However, automatic joint calibration based on existing calibration tool libraries has high requirements for the distance between the calibration board and the camera, and the measurement of the camera coordinate system position is not accurate, and the calibration is sensitive to strong light in the calibration environment, and the accuracy of the calibration is easily affected by environmental variables; the manual joint calibration method using fixed markers requires manual measurement of the distance between the laser radar and the calibration board, and the deviation is large, the accuracy of the measurement is not high, and the joint calibration requires high accuracy of the measured feature point quantity, and inaccurate coordinate data affects the accuracy of the joint calibration result, and the process is completely manual and cumbersome.

[0005] In summary, the existing camera and radar joint calibration method has the problems of complicated calibration process, high calibration requirements, and low calibration accuracy, which need to be solved. SUMMARY

[0006] Therefore, the present application provides a camera and radar joint calibration method and device to solve the above technical problems, and the technical solutions are as follows:

[0007] A camera and radar joint calibration method, comprising:

[0008] With the camera intrinsics and radar on the vehicle calibrated, when the calibration board is in at least three positions, the coordinates of the calibration board center in the pixel coordinate system are determined respectively, and the coordinates of the calibration board center in the radar coordinate system are determined respectively based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center.

[0009] Based on the coordinates of the calibration board center in the radar coordinate system and the pixel coordinate system at each position, a transformation relationship between the radar coordinate system and the pixel coordinate system is established for each position. The transformation relationship for each position includes a first transformation equation and a second transformation equation. The coordinates of the calibration board center in the pixel coordinate system include the first dimension coordinate and the second dimension coordinate. The first transformation equation includes the coordinates of the calibration board center in the radar coordinate system, the corresponding first dimension coordinate, and the camera's rotation and translation matrices. The second transformation equation includes the coordinates of the calibration board center in the radar coordinate system, the corresponding second dimension coordinate, and the camera's rotation and translation matrices.

[0010] The camera and radar are jointly calibrated based on the transformation relationship corresponding to each position, and the camera's rotation matrix and translation matrix are obtained.

[0011] Optionally, at least three locations are within a dense area of ​​point cloud wiring within the camera's field of view.

[0012] Optionally, when the calibration board is in at least three positions, the coordinates of the calibration board center in the pixel coordinate system are determined respectively, and the coordinates of the calibration board center in the radar coordinate system are determined respectively based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center, including:

[0013] When the calibration board is in a certain position, determine the coordinates of the center of the calibration board in the pixel coordinate system, and determine the coordinates of the center of the calibration board in the radar coordinate system based on the point cloud information of the center of the calibration board and / or the point cloud information of the surrounding points of the center of the calibration board.

[0014] If the number of times the calibration board moves is less than the preset total number of moves, after the calibration board moves to another position, the process returns to determine the coordinates of the calibration board center in the pixel coordinate system. Based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center, the coordinates of the calibration board center in the radar coordinate system are determined. The initial number of moves is 0, the preset total number of moves is greater than or equal to 2, and the position of the calibration board after each move is different.

[0015] Optionally, the coordinates of the calibration board center in the radar coordinate system are determined based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center, including:

[0016] Given point cloud information at the center of the calibration board, determine the coordinates of the center of the calibration board in the radar coordinate system based on the point cloud information at the center of the calibration board.

[0017] In the absence of point cloud information at the center of the calibration board, the coordinates of the center of the calibration board in the radar coordinate system are determined based on the point cloud information of the surrounding points.

[0018] Optional, the process for determining the center and surrounding points of the calibration plate includes:

[0019] Calculate the deflection angle of the center point of the target area containing the calibration board in the top view of the point cloud relative to the preset coordinate axis, where the preset coordinate axis points directly in front of the radar;

[0020] Rotate the top view of the point cloud according to the deflection angle so that the center point of the target area is directly in front of the radar, and use the rotated image as the front view of the point cloud of the target area.

[0021] Determine the center of the calibration board from the front view of the point cloud of the target area;

[0022] In the absence of point cloud information at the center of the calibration board, determine one or more points around the center of the calibration board that are closest to the center of the calibration board from the front view of the point cloud of the target area.

[0023] Optionally, the point cloud information of the surrounding points is point cloud coordinates;

[0024] Based on the point cloud information of the points surrounding the center of the calibration board, determine the coordinates of the center of the calibration board in the radar coordinate system, including:

[0025] If the points surrounding the center of the calibration board contain a single point, the point cloud coordinates of that point are determined as the coordinates of the center of the calibration board in the radar coordinate system.

[0026] When the center of the calibration board contains multiple points, calculate the average value of the point cloud coordinates of the multiple points, and determine the average value as the coordinates of the center of the calibration board in the radar coordinate system.

[0027] Optionally, the preset total number of times is any value between 8 and 11.

[0028] Optionally, at least three positions should be located within a distance of 4 to 6 meters from the front of the vehicle.

[0029] Optional, also includes:

[0030] If the obtained rotation and translation matrices are accurate, the coordinates used when obtaining the extrinsic parameters are stored in a preset file;

[0031] If the obtained rotation and translation matrices are inaccurate, after adjusting the position of the calibration board, return to the execution state where the calibration board is in at least three positions, determine the coordinates of the calibration board center in the pixel coordinate system, and determine the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center.

[0032] A combined calibration device for a camera and radar, comprising:

[0033] The coordinate determination unit is used to determine the coordinates of the calibration board center in the pixel coordinate system when the calibration board is in at least three positions, provided that the camera intrinsic parameters and radar on the vehicle are both calibrated; and to determine the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center.

[0034] The transformation relationship determination unit is used to establish the transformation relationship between the radar coordinate system and the pixel coordinate system for each position based on the coordinates of the center of the calibration board in the radar coordinate system and the pixel coordinate system respectively when the calibration board is in each position. The transformation relationship for each position includes a first transformation equation and a second transformation equation. The coordinates of the center of the calibration board in the pixel coordinate system include a first-dimensional coordinate and a second-dimensional coordinate. The first transformation equation includes the coordinates of the center of the calibration board in the radar coordinate system, the corresponding first-dimensional coordinate, and the rotation and translation matrices of the camera. The second transformation equation includes the coordinates of the center of the calibration board in the radar coordinate system, the corresponding second-dimensional coordinate, and the rotation and translation matrices of the camera.

[0035] The joint calibration unit is used to perform joint calibration of the camera and radar according to the transformation relationship corresponding to each position, so as to obtain the camera's rotation matrix and translation matrix.

[0036] As can be seen from the above technical solutions, the joint calibration method for camera and radar provided in this application, after calibrating the intrinsic parameters of the camera and radar on the vehicle, performs joint calibration of the camera and radar, resulting in stronger stability, stronger generalization of the calibration process, and stronger generalizability. Considering that the rotation matrix is ​​an orthogonal matrix with unit vectors in each row and column and pairwise orthogonal, with a rank of 3, and the translation matrix has 3 unknowns in its corresponding translation vector, this application needs to solve 6 unknowns in the joint calibration. Since the transformation relationship established when the calibration plate is in one position includes two transformation equations, this application can perform joint calibration by placing the calibration plate in at least three positions to obtain the camera's rotation and translation matrices. This application only needs to adjust the position of the calibration plate to achieve joint calibration. The calibration process does not require manual measurement of horizontal and vertical offset distances, avoiding errors caused by manual calibration and improving calibration accuracy and efficiency. At the same time, joint calibration based on the transformation relationship when the calibration plate is in at least three positions improves the accuracy of the calibration results. Furthermore, the calibration process of this application is simple and easy to operate, and there are no requirements on the distance between the camera and the calibration board. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 A schematic flowchart illustrating a joint calibration method for a camera and radar provided in an embodiment of this application;

[0039] Figure 2 A schematic diagram showing the location of the calibration plate provided in the embodiments of this application;

[0040] Figure 3 A schematic diagram of a point cloud top view provided for an embodiment of this application;

[0041] Figure 4 A schematic diagram of a target area and a deflection angle provided for an embodiment of this application;

[0042] Figure 5 A schematic diagram of a front view of a point cloud of a target area provided in an embodiment of this application;

[0043] Figure 6 A schematic diagram of the calibration process provided in the embodiments of this application;

[0044] Figure 7 A schematic diagram of the structure of a combined camera and radar calibration device provided in an embodiment of this application;

[0045] Figure 8 This is a hardware structure block diagram of a camera and radar joint calibration device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] This application provides a joint calibration method for cameras and radar, which can be applied to intelligent vehicles equipped with camera and radar perception systems. The following embodiments will provide a detailed description of the joint calibration method for cameras and radar provided in this application.

[0048] Please see Figure 1 The diagram illustrates a flowchart of a joint calibration method for a camera and radar provided in an embodiment of this application. This joint calibration method may include:

[0049] Step S101: With the camera intrinsics and radar on the vehicle both calibrated, when the calibration board is in at least three positions, determine the coordinates of the calibration board center in the pixel coordinate system, and determine the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center.

[0050] Specifically, this application requires first calibrating the camera's internal parameters and radar on the vehicle (it is worth noting that the relative positions of the radar and camera on the vehicle remain fixed). After completing the calibration of the camera's internal parameters and radar, the camera and radar are then jointly calibrated.

[0051] In this application, camera intrinsic parameter calibration is the determination of f. x f y u0 and v0, where (u0, v0) are the coordinates of the image center point in the pixel coordinate system, f x and f y These represent the distances in the x and y directions of the pixel coordinate system, respectively, i.e., f. x =F / dx, f y =F / dy, where F represents the focal length, and dx and dy represent the actual physical length (in meters) of one pixel in the x and y directions, respectively.

[0052] Those skilled in the art should understand that the principle of joint camera and radar calibration is to solve the transformation relationship between the radar coordinate system and the pixel coordinate system for the same point (i.e., feature point). The relationship between the radar coordinate system and the pixel coordinate system is as follows:

[0053]

[0054] In the formula, (u, v) represent the coordinates of the feature point in the pixel coordinate system, (X... w Y w Z w Z represents the coordinates of the feature point in the radar coordinate system. c The scaling factor cannot be equal to 0. It represents the distance from the center of the image coordinate system (i.e., the optical center) to the origin of the camera coordinate system. R and T are camera extrinsic parameters. R is a 3×3 rotation matrix and T is a 3×1 translation matrix.

[0055] It should be noted that the difference between the pixel coordinate system and the camera coordinate system is that the pixel coordinate system takes the center of the image as its origin, while the camera coordinate system takes the top left corner of the image as its origin.

[0056] From formula (1), in the camera intrinsic parameter f x f y Given u0 and v0, we only need to determine the coordinates of multiple feature points in the pixel coordinate system and the radar coordinate system respectively, and then substitute them into formula (1) to obtain the camera's external parameter matrix, which is also the camera's rotation matrix R and translation matrix T.

[0057] In this embodiment, the center of the calibration board can be used as a feature point. Considering that the rotation matrix is ​​an orthogonal matrix in which each row and column is a unit vector and each pair of vectors is orthogonal, and its rank is 3, while the translation vector corresponding to the translation matrix has 3 unknowns, this application needs to solve 6 unknowns in the joint calibration. Since the pixel coordinate system is a two-dimensional coordinate system, it can be easily understood by combining formula (1) that two transformation equations can be established for each feature point (i.e., the center of the calibration board). Therefore, this application can place the calibration board in at least three positions, and determine the coordinates of the center of the calibration board in the radar coordinate system and the pixel coordinate system at each position.

[0058] It is worth noting that in the pixel coordinate system, each pixel of an image has corresponding coordinates. Therefore, by taking a picture of the calibration board with a camera, this application can easily determine the coordinates of the calibration board center in the pixel coordinate system from the captured image. However, for radar, since point cloud information is the information of a laser beam emitted at a certain angular velocity hitting an obstacle and returning, it is not as dense as pixel information. It may or may not have point cloud information of the calibration board center. Therefore, this application can determine the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center.

[0059] Step S102: Based on the coordinates of the center of the calibration board in the radar coordinate system and the pixel coordinate system when the calibration board is in each position, establish the transformation relationship between the radar coordinate system and the pixel coordinate system corresponding to each position.

[0060] Specifically, after obtaining the coordinates of the center of the calibration plate in the radar coordinate system and the pixel coordinate system respectively when the calibration plate is in each position in the aforementioned steps, the obtained coordinates can be substituted into the above formula (1) to obtain the transformation relationship between the radar coordinate system and the pixel coordinate system corresponding to each position.

[0061] Here, the transformation relationship corresponding to each position includes a first transformation equation and a second transformation equation. The coordinates of the calibration board center in the pixel coordinate system include the first dimension coordinates and the second dimension coordinates. The first transformation equation includes the coordinates of the calibration board center in the radar coordinate system, the corresponding first dimension coordinates, and the camera's rotation and translation matrices. The second transformation equation includes the coordinates of the calibration board center in the radar coordinate system, the corresponding second dimension coordinates, and the camera's rotation and translation matrices.

[0062] For example, when the feature point is at a certain position, the coordinates of the calibration board center P in the pixel coordinate system are (x1, y1), and the coordinates of P in the radar coordinate system are (x0, y0, z0). If the transformation relationship in formula (1) is represented by k (k contains 6 unknowns), then the two transformation equations are: x1 = k(x0, y0, z0) and y1 = k(x0, y0, z0).

[0063] Step S103: Perform joint calibration of the camera and radar according to the transformation relationship corresponding to each position to obtain the camera's rotation matrix and translation matrix.

[0064] Specifically, this application allows the calibration board to be in at least three positions, thus obtaining at least six transformation equations. By iteratively solving these six transformation equations, the specific values ​​of the six unknowns can be obtained, which in turn yields the camera's rotation and translation matrices, completing the joint calibration of the camera and radar.

[0065] The joint calibration method for cameras and radar provided in this application, after calibrating the intrinsic parameters of both the camera and radar on the vehicle, further enhances stability, generalizability, and scalability. Considering that the rotation matrix is ​​an orthogonal matrix with unit vectors in each row and column, and each pair of vectors is pairwise orthogonal, with a rank of 3, while the translation matrix corresponds to a translation vector with three unknowns, this application requires solving for six unknowns in the joint calibration. Since the transformation relationship established when the calibration board is in one position includes two transformation equations, this application allows joint calibration to be performed with the calibration board in at least three positions to obtain the camera's rotation and translation matrices. This application only needs to adjust the position of the calibration board to achieve joint calibration, eliminating the need for manual measurement of horizontal and vertical offset distances, avoiding errors caused by manual calibration, and improving calibration accuracy and efficiency. Furthermore, joint calibration based on the transformation relationship when the calibration board is in at least three positions improves the accuracy of the calibration results. In addition, the calibration process of this application is simple and easy to operate, and there are no requirements regarding the distance between the camera and the calibration board.

[0066] An embodiment of this application describes the process in step S101 described above, which involves "determining the coordinates of the calibration board center in the pixel coordinate system when the calibration board is in at least three positions, and determining the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center."

[0067] It is understandable that point clouds are denser in some areas and sparser in others. In order to have as much point cloud information as possible in the center of the calibration board and / or the area surrounding the center of the calibration board, it is preferable to place the calibration board in a dense point cloud area within the camera's field of view when setting the location of the calibration board. That is, at least the above three locations are all located in a dense point cloud area within the camera's field of view.

[0068] For example, see Figure 2 The schematic diagram showing the location of the calibration plate indicates that the distance between the calibration plate and the front of the vehicle is 4 to 6 meters. That is, the distance between the above at least three locations and the front of the vehicle is within the range of 4 to 6 meters, and the height of the calibration plate is in the dense radar beam area.

[0069] Optional, such as Figure 2 As shown, this application can place 4 to 6 positions at each distance with the radar position as the center and a certain distance as the radius, for a total of 12 positions, to ensure that the calibration board at each position is within the camera's field of view and in an area with dense point cloud information.

[0070] After determining the location of the calibration board, step S101 can be implemented in at least two of the following ways.

[0071] In one possible implementation, it can be, for example... Figure 2 A calibration board is set at each of the at least three locations shown. Then, for each calibration board, the coordinates of the center of the calibration board in the radar coordinate system and the coordinates in the pixel coordinate system are determined respectively.

[0072] In another possible implementation, considering that calibration plates are set at at least three locations, a large number of calibration plates are required, resulting in high costs. To reduce costs, this embodiment can achieve this by moving the positions of the calibration plates.

[0073] Specifically, this application can determine the coordinates of the calibration board center in the pixel coordinate system when the calibration board is in a certain position, and determine the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center.

[0074] If the number of times the calibration board moves is less than the preset total number of moves, after the calibration board moves to another position, the process returns to determine the coordinates of the calibration board center in the pixel coordinate system. Based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center, the coordinates of the calibration board center in the radar coordinate system are determined. The initial number of moves is 0, the total number of moves is greater than or equal to 2, and the position of the calibration board is different after each move.

[0075] For example, if the preset total number of moves is 2, when the calibration board is in position 1 that meets the above position requirements, the coordinates of the calibration board center in the radar coordinate system and the pixel coordinate system are determined to obtain a set of coordinates; the initial number of moves is 0, which is less than the preset total number of moves 2, so the calibration board is moved to position 2 that meets the above position requirements, and the coordinates of the calibration board center in the radar coordinate system and the pixel coordinate system are determined again to obtain another set of coordinates; the current number of moves is 1, which is less than the preset total number of moves 2, so the calibration board is moved to position 3 that meets the above position requirements, and the coordinates of the calibration board center in the radar coordinate system and the pixel coordinate system are determined again to obtain yet another set of coordinates; the current number of moves is 2, which is equal to the preset total number of moves 2, at which point the process ends.

[0076] In this application, through multiple experiments verified by the inventors, the accuracy of marking is highest when the number of feature points is selected between 9 and 12. Preferably, the preset total number of times is any value between 8 and 11. Specifically, when the preset total number of times is 8, this application can determine 9 sets of coordinates, and then establish a transformation relationship based on the 9 sets of coordinates and perform joint calibration; when the preset total number of times is 9, this application can determine 10 sets of coordinates, and then establish a transformation relationship based on the 10 sets of coordinates and perform joint calibration; when the preset total number of times is 10, this application can determine 11 sets of coordinates, and then establish a transformation relationship based on the 11 sets of coordinates and perform joint calibration; when the preset total number of times is 11, this application can determine 12 sets of coordinates, and then establish a transformation relationship based on the 12 sets of coordinates and perform joint calibration.

[0077] In an optional embodiment, the process of "determining the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the calibration board center and / or the point cloud information of the surrounding points of the calibration board center" includes: if there is point cloud information at the calibration board center, determining the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the calibration board center; if there is no point cloud information at the calibration board center, determining the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the surrounding points of the calibration board center.

[0078] Of course, the process of determining the coordinates of the calibration board center in the radar coordinate system provided in this embodiment is only an example. Other implementation methods are also possible. For example, if there is point cloud information at the center of the calibration board, the coordinates of the calibration board center in the radar coordinate system can be determined based on the point cloud information of the calibration board center and the point cloud information of the surrounding points, etc.

[0079] Optionally, the point cloud information of the surrounding points mentioned above is point cloud coordinates. The process of "determining the coordinates of the calibration board center in the radar coordinate system based on the point cloud information of the surrounding points of the calibration board center" may include: if the surrounding points of the calibration board center include one point, determining the point cloud coordinates of that point as the coordinates of the calibration board center in the radar coordinate system; if the surrounding points of the calibration board center include multiple points, calculating the average value of the point cloud coordinates of the multiple points, and determining the calculated average value as the coordinates of the calibration board center in the radar coordinate system.

[0080] Specifically, when determining the points around the center of the calibration board, there may be only one point around the center of the calibration board with point cloud coordinates. In this case, the point cloud coordinates of that point are determined as the coordinates of the center of the calibration board in the radar coordinate system. There may also be multiple points around the center of the calibration board with point cloud coordinates. In this case, the average value of the point cloud coordinates of multiple points can be calculated, and the calculated average value is determined as the coordinates of the center of the calibration board in the radar coordinate system.

[0081] Another embodiment of this application describes the process of determining the center and surrounding points of the calibration plate mentioned in the foregoing embodiments.

[0082] Optionally, the process of determining the center and surrounding points of the calibration plate may include:

[0083] Step S11: Calculate the deflection angle of the center point of the target area containing the calibration board in the top view of the point cloud relative to the preset coordinate axis, wherein the preset coordinate axis points directly in front of the radar.

[0084] Here, a point cloud top view refers to a top view of the point cloud information collected by the radar over an area containing the calibration board; for example, see [link to relevant documentation]. Figure 3 The diagram shown is a top view of the point cloud. Figure 3 The white color represents point cloud information.

[0085] This application allows users to select a region of interest (ROI) including the calibration plate in the point cloud top-view interface. Irrelevant point clouds are then removed based on this ROI, enabling faster feature point identification. For ease of subsequent description, the selected ROI is defined as the target region.

[0086] It is understandable that the target area may not be directly in front of the radar. In order to facilitate the selection of the calibration board center in this embodiment, the deflection angle θ of the center point of the target area containing the calibration board in the point cloud top view relative to the preset coordinate axis can be calculated first.

[0087] See Figure 4 The diagram shown illustrates a target area and deflection angle according to an embodiment of this application. The target area is the region indicated by the white box. Figure 3 The top view shown indicates the target area. After selection, mark the deflection angle θ in the point cloud top view to obtain... Figure 4 .

[0088] This application can be based on Figure 4 Calculate the deflection angle θ of the center point of the target area plane relative to the x-axis of the plane coordinate system in the radar coordinate system.

[0089] Step S12: Rotate the top view of the point cloud according to the deflection angle so that the center point of the target area is located directly in front of the radar, and use the rotated image as the front view of the point cloud of the target area.

[0090] Specifically, the target area is deflected by θ based on the front of the radar. In order to see the calibration board more intuitively and find the feature points quickly, the top view of the point cloud can be rotated according to the deflection angle so that the center point of the target area is located in front of the radar.

[0091] For example, see Figure 5 The image shown is a schematic diagram of the front view of the point cloud of the target area. This diagram is derived from... Figure 4Obtained by rotating according to the deflection angle θ.

[0092] Step S13: Determine the center of the calibration plate from the front view of the point cloud of the target area.

[0093] Specifically, this application allows the center of the calibration plate to be selected as a feature point in the interface of the front view of the point cloud of the target area.

[0094] Step S14: If there is no point cloud information at the center of the calibration board, determine one or more points around the center of the calibration board that are closest to the center of the calibration board from the front view of the point cloud of the target area.

[0095] Specifically, since the point cloud information is relatively sparse, if there is no point cloud information for the center point determined in the previous steps, this step can be used to select one or more points around the center of the calibration board that are closest to the center of the calibration board.

[0096] To enable those skilled in the art to better understand this application, the following optional embodiment, in conjunction with... Figure 6 The calibration process provided in this application is described.

[0097] The calibration prerequisite for this application is the completion of camera intrinsic parameter calibration and radar calibration.

[0098] Tools required: calibration plate and tripod for fixing the calibration plate.

[0099] Step S21: Place the calibration plate.

[0100] Specifically, according to Figure 2 The calibration board is placed in the dense point cloud area at the indicated location. For example, the calibration board is placed 4 to 6 meters away from the front of the vehicle, at a height in the dense radar beam area. With the radar position as the center and a certain distance as the radius, 4 to 6 positions are placed at each distance, for a total of 12 positions. This ensures that the calibration board is within the image display range and the dense point cloud information area at each position.

[0101] Step S22: Open the camera and radar calibration interfaces and synchronize their times to begin calibration.

[0102] Among them, the radar calibration interface refers to the point cloud top view interface.

[0103] Step S23: Select multiple feature points on the radar calibration interface.

[0104] Specifically, the selection process for each feature point includes: a manual selection of the target area, including the calibration plate, on the top view of the point cloud. After the selection is completed, this application can obtain the selected target area on the top view of the point cloud, calculate the deflection angle of the center point of the target area containing the calibration plate relative to the preset coordinate axis, and rotate the top view of the point cloud according to the deflection angle to obtain the front view of the point cloud of the target area. Then, the center of the calibration plate is selected as the feature point in the front view of the point cloud of the target area.

[0105] Step S24: Solve for the calibration results.

[0106] Specifically, for each feature point selected in the aforementioned steps, this step can determine the coordinates of that feature point in both the pixel coordinate system and the radar coordinate system.

[0107] The process of determining the coordinates of the feature point in the radar coordinate system includes: if there is point cloud information at the center of the calibration board, this step can determine the coordinates of the center of the calibration board in the radar coordinate system based on the point cloud information at the center of the calibration board; if there is no point cloud information at the center of the calibration board, select the surrounding points of the center of the calibration board, and determine the coordinates of the center of the calibration board in the radar coordinate system based on the point cloud information of the surrounding points of the center of the calibration board.

[0108] The process of determining the coordinates of the feature point in the pixel coordinate system includes: calibrating the image interface, selecting the center of the calibration plate as the feature point, and recording the coordinates of the feature point in the pixel coordinate system.

[0109] Optionally, this application can obtain 9 to 12 feature points by moving the position of the calibration board or setting multiple calibration boards at multiple positions in the dense point cloud area.

[0110] Optionally, the coordinates of each feature point obtained can be saved in both the radar coordinate system and the pixel coordinate system.

[0111] After obtaining the coordinates of each feature point in the radar coordinate system and the pixel coordinate system, this application can substitute the coordinates of each feature point in the radar coordinate system and the pixel coordinate system into formula (1) in turn, and complete the joint calibration of camera and radar based on iterative solution of camera extrinsic parameters.

[0112] Optionally, the joint calibration method for camera and radar provided in this application embodiment may further include the following step S25.

[0113] Step S25: If the calibration result is satisfactory, the feature points are automatically saved as a separate file. If the calibration is not satisfactory, the feature points are reselected for calibration.

[0114] Specifically, if the obtained rotation and translation matrices are accurate, this application stores the coordinates used when obtaining the extrinsic parameter matrix into a preset file; if the obtained rotation and translation matrices are inaccurate, after adjusting the position of the calibration board, it returns to execute the following steps: when the calibration board is in at least three positions, the coordinates of the calibration board center in the pixel coordinate system are determined respectively, and the coordinates of the calibration board center in the radar coordinate system are determined respectively based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center.

[0115] In summary, the joint calibration method for cameras and radar provided in this application mainly relies on the calibration parameters of lidar, vehicle, and camera to calibrate and solve the coordinate system transformation relationship, which has the following beneficial effects: First, the calibration process does not require manual measurement of horizontal and vertical offset distances, eliminating the tedious process of marking and measuring ground feature points in existing schemes, thus improving calibration accuracy and efficiency; Second, the calibration process uses the average of multiple sets of feature points to solve the problem, resulting in high accuracy of the calibration results; Third, the calibration process is based on camera intrinsic parameters and radar calibration, exhibiting strong stability, strong generalization, and high scalability.

[0116] This application also provides a joint calibration device for a camera and radar. The joint calibration device for a camera and radar provided in this application is described below. The joint calibration device for a camera and radar described below can be referred to in correspondence with the joint calibration method for a camera and radar described above.

[0117] Please see Figure 7 The diagram shows a schematic representation of the combined camera and radar calibration device provided in an embodiment of this application. Figure 7 As shown, the joint calibration device for the camera and radar may include: a coordinate determination unit 701, a transformation relationship determination unit 702, and a joint calibration unit 703.

[0118] The coordinate determination unit 701 is used to determine the coordinates of the center of the calibration board in the pixel coordinate system when the calibration board is in at least three positions, provided that the camera intrinsic parameters and radar on the vehicle are both calibrated, and to determine the coordinates of the center of the calibration board in the radar coordinate system based on the point cloud information of the center of the calibration board and / or the point cloud information of the points surrounding the center of the calibration board.

[0119] The transformation relationship determination unit 702 is used to establish a transformation relationship between the radar coordinate system and the pixel coordinate system corresponding to each position based on the coordinates of the center of the calibration board in the radar coordinate system and the pixel coordinate system respectively when the calibration board is in each position. The transformation relationship corresponding to each position includes a first transformation equation and a second transformation equation. The coordinates of the center of the calibration board in the pixel coordinate system include a first dimension coordinate and a second dimension coordinate. The first transformation equation includes the coordinates of the center of the calibration board in the radar coordinate system, the corresponding first dimension coordinate, and the rotation matrix and translation matrix of the camera. The second transformation equation includes the coordinates of the center of the calibration board in the radar coordinate system, the corresponding second dimension coordinate, and the rotation matrix and translation matrix of the camera.

[0120] The joint calibration unit 703 is used to perform joint calibration of the camera and the radar according to the transformation relationship corresponding to each of the positions, so as to obtain the rotation matrix and translation matrix of the camera.

[0121] The working principle of the camera and radar joint calibration device provided in this application is the same as that of the aforementioned camera and radar joint calibration method. For details, please refer to the foregoing description, which will not be repeated here.

[0122] This application also provides a joint calibration device for a camera and radar. Optionally, Figure 8 The hardware structure block diagram of the joint calibration device for cameras and radar is shown, with reference to... Figure 8 The hardware structure of the joint calibration device for the camera and radar may include: at least one processor 801, at least one communication interface 802, at least one memory 803 and at least one communication bus 804.

[0123] In this embodiment, the number of processor 801, communication interface 802, memory 803, and communication bus 804 is at least one, and processor 801, communication interface 802, and memory 803 communicate with each other through communication bus 804.

[0124] The processor 801 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.

[0125] The memory 803 may include high-speed RAM, or it may also include non-volatile memory, such as at least one disk storage device;

[0126] The memory 803 stores a program, and the processor 801 can call the program stored in the memory 803. The program is used for:

[0127] With the camera intrinsics and radar on the vehicle calibrated, when the calibration board is in at least three positions, the coordinates of the calibration board center in the pixel coordinate system are determined respectively, and the coordinates of the calibration board center in the radar coordinate system are determined respectively based on the point cloud information of the calibration board center and / or the point cloud information of the points surrounding the calibration board center.

[0128] Based on the coordinates of the calibration board center in the radar coordinate system and the pixel coordinate system at each position, a transformation relationship between the radar coordinate system and the pixel coordinate system is established for each position. The transformation relationship for each position includes a first transformation equation and a second transformation equation. The coordinates of the calibration board center in the pixel coordinate system include the first dimension coordinate and the second dimension coordinate. The first transformation equation includes the coordinates of the calibration board center in the radar coordinate system, the corresponding first dimension coordinate, and the camera's rotation and translation matrices. The second transformation equation includes the coordinates of the calibration board center in the radar coordinate system, the corresponding second dimension coordinate, and the camera's rotation and translation matrices.

[0129] The camera and radar are jointly calibrated based on the transformation relationship corresponding to each position, and the camera's rotation matrix and translation matrix are obtained.

[0130] Optionally, the refined and extended functions of the program can be found in the description above.

[0131] This application also provides a readable storage medium storing a computer program thereon, which, when executed by a processor, implements the joint calibration method for cameras and radar as described above.

[0132] Optionally, the refined and extended functions of the program can be found in the description above.

[0133] Finally, it should be noted that in this document, relational terms such as "second" and "etc." are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0135] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for joint calibration of a camera and a radar, characterized in that, The application relates to a camera and radar joint calibration method and device. In the case that camera intrinsic parameters and radar are calibrated on a vehicle, when a calibration board is in at least three positions, coordinates of a center of the calibration board in a pixel coordinate system are determined respectively, and coordinates of the center of the calibration board in a radar coordinate system are determined respectively according to point cloud information of the center of the calibration board and / or point cloud information of surrounding points of the center of the calibration board; According to the coordinates of the center of the calibration board in the radar coordinate system and the pixel coordinate system respectively when the calibration board is in each of the positions, a conversion relationship between the radar coordinate system and the pixel coordinate system corresponding to each of the positions is established, wherein the conversion relationship corresponding to each of the positions comprises a first conversion equation and a second conversion equation, the coordinates of the center of the calibration board in the pixel coordinate system comprise a first dimension coordinate and a second dimension coordinate, the first conversion equation comprises the coordinates of the center of the calibration board in the radar coordinate system, the corresponding first dimension coordinate, and a rotation matrix and a translation matrix of the camera, and the second conversion equation comprises the coordinates of the center of the calibration board in the radar coordinate system, the corresponding second dimension coordinate, and the rotation matrix and the translation matrix of the camera; The camera and the radar are jointly calibrated according to the conversion relationship corresponding to each of the positions, so that the rotation matrix and the translation matrix of the camera are obtained.

2. The camera and radar joint calibration method of claim 1, wherein, The at least three positions are located in a point cloud beam dense area within a visible range of the camera.

3. The camera and radar joint calibration method of claim 2, wherein, The method comprises the following steps: When the calibration board is in a position, the coordinates of the center of the calibration board in the pixel coordinate system are determined, and the coordinates of the center of the calibration board in the radar coordinate system are determined according to the point cloud information of the center of the calibration board and / or the point cloud information of surrounding points of the center of the calibration board; When the moving times of the calibration board are less than a preset total times, after the calibration board moves to another position, the coordinates of the center of the calibration board in the pixel coordinate system are determined, and the coordinates of the center of the calibration board in the radar coordinate system are determined according to the point cloud information of the center of the calibration board and / or the point cloud information of surrounding points of the center of the calibration board, wherein the initial moving times are 0, the preset total times are greater than or equal to 2, and the positions of the calibration board after each moving are different.

4. The camera and radar joint calibration method of claim 1, wherein, When the center of the calibration board has point cloud information, the coordinates of the center of the calibration board in the radar coordinate system are determined according to the point cloud information of the center of the calibration board; When the center of the calibration board has no point cloud information, the coordinates of the center of the calibration board in the radar coordinate system are determined according to the point cloud information of surrounding points of the center of the calibration board. ​ 5. The camera and radar joint calibration method of claim 4, wherein, The determination process of the calibration board center and the surrounding points comprises: calculating a deflection angle of a center point of a target region containing the calibration board in a point cloud overhead view relative to a preset coordinate axis, wherein the preset coordinate axis points to the front of the radar; rotating the point cloud overhead view according to the deflection angle, so that the center point of the target region is located in the front of the radar, and taking the rotated view as a target region point cloud front view; determining the calibration board center from the target region point cloud front view; in the case that there is no point cloud information of the calibration board center, determining one or more points closest to the calibration board center from the target region point cloud front view.

6. The camera and radar joint calibration method of claim 4, wherein, The point cloud information of the surrounding points of the calibration board center is point cloud coordinates; The determination of the coordinates of the calibration board center in the radar coordinate system according to the point cloud information of the surrounding points of the calibration board center comprises: in the case that the surrounding points of the calibration board center contain one point, determining the point cloud coordinates of the point as the coordinates of the calibration board center in the radar coordinate system; in the case that the surrounding points of the calibration board center contain multiple points, calculating the average of the point cloud coordinates of the multiple points, and determining the calculated average as the coordinates of the calibration board center in the radar coordinate system.

7. The camera and radar joint calibration method of claim 3, wherein, The preset total number is any value in 8-11.

8. The camera and radar joint calibration method of claim 2, wherein, The distance between the at least three positions and the front of the vehicle is in the range of 4-6 meters.

9. The camera and radar joint calibration method of claim 1, wherein, Further comprising: in the case that the obtained rotation matrix and translation matrix are accurate, storing the coordinates used to obtain the extrinsic matrix into a preset file; in the case that the obtained rotation matrix and translation matrix are not accurate, after adjusting the position of the calibration board, returning to execute the determination of the coordinates of the calibration board center in the pixel coordinate system and the determination of the coordinates of the calibration board center in the radar coordinate system according to the point cloud information of the calibration board center and / or the point cloud information of the surrounding points of the calibration board center.

10. A combined camera and radar calibration apparatus, characterized by, Comprise: a coordinate determination unit, configured to, in the case that the camera intrinsic parameter and the radar are both calibrated, determine the coordinates of the calibration board center in the pixel coordinate system and the coordinates of the calibration board center in the radar coordinate system according to the point cloud information of the calibration board center and / or the point cloud information of the surrounding points of the calibration board center when the calibration board is in at least three positions. The conversion relationship determining unit is configured to establish a conversion relationship between the radar coordinate system and the pixel coordinate system corresponding to each of the positions according to the coordinates of the center of the calibration board in the radar coordinate system and the pixel coordinate system respectively when the calibration board is in each of the positions, wherein the conversion relationship corresponding to each of the positions comprises a first conversion equation and a second conversion equation, the coordinates of the center of the calibration board in the pixel coordinate system comprise a first dimension coordinate and a second dimension coordinate, the first conversion equation comprises the coordinates of the center of the calibration board in the radar coordinate system, the corresponding first dimension coordinate, and the rotation matrix and the translation matrix of the camera, and the second conversion equation comprises the coordinates of the center of the calibration board in the radar coordinate system, the corresponding second dimension coordinate, and the rotation matrix and the translation matrix of the camera. The joint calibration unit is configured to jointly calibrate the camera and the radar according to the conversion relationship corresponding to each of the positions to obtain the rotation matrix and the translation matrix of the camera.

Images