Calibration method and device

Abstract

The invention discloses a calibration method and device, relates to the technical field of intelligent driving, and is used for calibrating the geometrical relationship among a camera, a laser radar and a vehicle and reducing the calibration complexity. Through the calibration objects of different types on the calibration target, the external parameters of the laser radar and the camera relative to the vehicle are jointly calibrated, the geometrical relationship between the laser radar and the camera does not need to be calibrated, and then the geometrical relationship between the laser radar, the camera and the vehicle does not need to be further calibrated, so that the calibration complexity can be reduced, and the calibration precision is improved. For example, a reflector or a calibration hole (which can be chamfered) is used as a calibration object of a laser radar, and a pattern is set as a calibration object of a camera, so that edge points and / or edge lines are extracted conveniently, and coordinates in a calibration area are obtained. The laser radar and the camera are further calibrated through the coordinates of the calibration area in the world coordinate system, and the complexity is reduced.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and particularly relates to a calibration method and device. Background Art

[0002] Vehicles are increasingly commonly equipped with advanced driver assistance systems, such as intelligent parking, lane keeping assistance, and adaptive cruise control. These systems typically rely on cameras and lidar to sense the environment and obstacles around the vehicle. To ensure the accuracy and reliability of these systems, precise calibration of the cameras and lidar is required.

[0003] Currently, vehicles are equipped with cameras and lidar for data fusion to obtain a perception of the environment around the vehicle. The camera provides visual information, and the lidar provides distance and three-dimensional information. To fuse the information of the camera and lidar in a coordinate system for processing, calibration of the camera and lidar is required. However, current calibration methods only obtain the geometric relationship between the camera and the lidar. But if the geometric relationship between the camera, lidar and the vehicle needs to be obtained, separate calibration is still required, and the calibration process is complex and the accuracy is low. Summary of the Invention

[0004] Embodiments of this application provide a calibration method and device for calibrating the geometric relationship between a camera, a lidar and a vehicle, and reducing the calibration complexity.

[0005] In a first aspect, embodiments of this application provide a calibration method, including: obtaining point cloud data of at least one calibration target collected by a lidar on a vehicle and image frame data of at least one calibration target collected by a camera on the vehicle; each calibration target in the at least one calibration target includes at least one first type of calibration object and at least one second type of calibration object; determining the coordinates of at least one first type of calibration object in the lidar coordinate system according to the point cloud data, and determining the coordinates of at least one second type of calibration object in the image coordinate system of the camera according to the image frame data; calibrating a first external parameter of the lidar relative to the vehicle according to the position coordinates of at least one first type of calibration object in the world coordinate system and the position coordinates of at least one first type of calibration object in the lidar coordinate system, and calibrating a second external parameter of the camera relative to the vehicle according to the position coordinates of at least one second type of calibration object in the world coordinate system and the position coordinates of at least one second type of calibration object in the image coordinate system of the camera; wherein, the world coordinate system is established with the vehicle as a reference.

[0006] This application uses the above calibration scheme to directly calibrate the extrinsic parameters of the lidar and the camera relative to the vehicle by calibrating different types of calibration objects on the calibration target, without first calibrating the geometric relationship between the lidar and the camera and then further calibrating the geometric relationship between the lidar, the camera and the vehicle, which can reduce the calibration complexity and improve the calibration accuracy.

[0007] In a possible implementation, the calibration area of the first type of calibration object is a set geometric shape.

[0008] In a possible implementation, the calibration area is a mirror, or the calibration area is a calibration hole. In this implementation, the lidar is calibrated using a mirror or a calibration hole, which is convenient for extracting the edge points of the calibration area, so as to obtain the coordinates of the calibration area in the radar coordinate system, and further calibrate the lidar through the coordinates of the calibration area in the world coordinate system, with relatively low complexity.

[0009] In a possible implementation, the mirror is circular, square or triangular.

[0010] In a possible implementation, the calibration hole is circular, square or triangular.

[0011] In a possible implementation, the inner wall of the calibration hole is chamfered. In the above implementation, the inner wall of the calibration hole is designed to be chamfered to facilitate the light rays of the lidar to pass through the hole, prevent them from reaching the inner wall of the hole, and make it easier to extract the edge points around the hole to achieve the calibration of the lidar.

[0012] In a possible implementation, the position coordinates of the first type of calibration object in the lidar coordinate system are the coordinates of the geometric center of the set geometric shape in the lidar coordinate system.

[0013] In a possible implementation, determining the coordinates of at least one first type of calibration object in the lidar coordinate system according to the point cloud data includes: obtaining the edge points of the set geometric shape on the first type of calibration object according to the point cloud data, and fitting the geometric center of the geometric shape according to the edge points.

[0014] In the above implementation, using the coordinates of the geometric center as the coordinates of the calibration object in the radar coordinate system is convenient for fitting through the edge points of the geometric shape and reduces the complexity.

[0015] In a possible implementation, determining the position coordinates of the at least one first type of calibration object in the lidar coordinate system according to the point cloud data includes:

[0016] Determining the point cloud data of the region of interest from the point cloud data, where the region of interest includes the first type of calibration object;

[0017] Determine the edge points with significant features in the point cloud data of the region of interest;

[0018] Map the edge points with significant features to the two-dimensional image coordinate system to obtain a two-dimensional image;

[0019] Perform template segmentation on the two-dimensional image to obtain at least one light cluster region, and the at least one light cluster region corresponds one-to-one to the at least one first type of calibration object; the light cluster region is used to indicate the interval range of the set geometric shape on the corresponding first type of calibration object in the two-dimensional image;

[0020] Perform non-maximum suppression on the at least one light cluster region respectively, and process the at least one light cluster region after non-maximum suppression to obtain the edge points of the set geometric shape on the at least one first type of calibration object;

[0021] According to the coordinates of the edge points of the set geometric shape on the at least one first type of calibration object in the lidar coordinate system, determine the coordinates of the geometric midpoint of the set geometric shape on the at least one first type of calibration object in the lidar coordinate system respectively.

[0022] In the above implementation, the coordinates of the geometric center are used as the coordinates of the calibration object in the radar coordinate system, which is convenient for fitting through the edge points of the geometric shape and reduces the complexity.

[0023] Processing the at least one light cluster region after non-maximum suppression may be to filter out the outlier points in each light cluster region after non-maximum suppression. Exemplarily, to process the at least one light cluster region after non-maximum suppression, a clustering algorithm may be used to filter out the outlier points in the light cluster region and leave the edge points to be fitted. Thereby improving the accuracy of the determined coordinates of the geometric center.

[0024] In a possible implementation manner, the number of the first type of calibration objects is N, where N is an integer greater than 1. Calibrating the first external parameter of the lidar relative to the vehicle according to the position coordinates of the at least one first type of calibration object in the world coordinate system and the position coordinates of the at least one first type of calibration object in the lidar coordinate system includes:

[0025] Based on the coordinates of the geometric centers of the N first type of calibration objects in the world coordinate system respectively, and the coordinates of the geometric centers of the N first type of calibration objects in the lidar coordinate system respectively, solve the first mapping equation to obtain the first external parameter of the lidar relative to the vehicle;

[0026] Among them, the first mapping equation is established based on the mapping relationship between the world coordinate system and the lidar coordinate system.

[0027] In the above implementation, the external parameters are solved by calculating the mapping equation, and the implementation is simple, which can reduce the complexity.

[0028] In a possible implementation, the method further includes:

[0029] After determining the first external parameters, based on the coordinates of the geometric centers of the N first-type calibration objects in the world coordinate system respectively, and the coordinates of the geometric centers of the N first-type calibration objects in the lidar coordinate system respectively, and using the first global cost function to optimize the first external parameters to obtain the optimized first external parameters;

[0030] The first global cost function is used to characterize the error between the coordinates of the geometric center of the first-type calibration object in the world coordinate system and the coordinates of the geometric center of the first-type calibration object in the lidar coordinate system transformed to the world coordinate system based on the first external parameters.

[0031] In the above implementation, after solving the external parameters by calculating the mapping equation, further optimization is performed to improve the accuracy of the calibrated external parameters.

[0032] In a possible implementation, the calibration area of the second-type calibration object is a set pattern, and the set pattern includes at least points and lines. For example, it consists of points and lines. At least one calibration point is included in the points of the set pattern. At least one calibration line is included in the lines of the set pattern.

[0033] In a possible implementation, determining the position coordinates of the at least one second-type calibration object in the image coordinate system of the camera according to the image frame data includes:

[0034] Using an edge detection algorithm to detect the image data of the set pattern from the image frame data;

[0035] Detecting at least one calibration point and / or at least one calibration line in the set pattern according to the image data of the second-type calibration object to obtain the coordinates of the at least one calibration point and / or at least one calibration line in the image coordinate system of the camera.

[0036] In a possible implementation, the calibration point is a corner point of the set pattern, and the calibration line is at least one edge line of the set pattern. Detecting the calibration point and / or calibration line in the set pattern according to the image data of the second-type calibration object includes:

[0037] Obtain multiple corner points and / or multiple edge lines in the set pattern according to the image data of the set pattern;

[0038] Determine descriptors of the multiple corner points and / or descriptors of the multiple edge lines;

[0039] According to the descriptors of the multiple edge points, screen at least one corner point as a calibration point from the multiple corner points, and / or, according to the descriptors of the multiple edge lines, screen at least one edge line as a calibration line from the multiple edge points, so as to obtain the coordinates of the at least one corner point as a calibration point and / or the at least one edge line as a calibration line in the image coordinate system of the camera.

[0040] The above solution uses corner points and one or several edge lines for calibration. Through the edge extraction method and the corner point extraction method, the implementation is relatively simple and the accuracy is relatively high.

[0041] In a possible implementation manner, the number of the second type of calibration objects is M, where M is an integer greater than 1. According to the position coordinates of the at least one second type of calibration object in the world coordinate system and the coordinates of the at least one second type of calibration object in the image coordinate system of the camera, calibrate the second external parameter of the camera relative to the vehicle, including:

[0042] Based on the coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second type of calibration objects in the world coordinate system, and the coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second type of calibration objects in the image coordinate system of the camera, solve the second mapping equation to obtain the second external parameter of the camera relative to the vehicle;

[0043] Wherein, the second mapping equation is established based on the mapping relationship between the world coordinate system and the image coordinate system of the camera.

[0044] In the above implementation manner, after solving the external parameter by solving the mapping equation, further optimization is performed to improve the accuracy of the calibrated external parameter.

[0045] In a possible implementation manner, the method further includes:

[0046] After determining the second external parameter, based on the coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second type of calibration objects in the world coordinate system, and the coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second type of calibration objects in the lidar coordinate system, and use the second global cost function to optimize the second external parameter to obtain the optimized second external parameter;

[0047] The second global cost function is used to characterize the error between the coordinates of the calibration marker in the image coordinate system of the camera and the coordinates of the calibration marker in the world coordinate system after being transformed to the image coordinate system using the second extrinsic parameter. The calibration marker includes at least one calibration point and / or at least one calibration line.

[0048] In the above implementation, after solving for the extrinsic parameters by solving the mapping equation, further optimization is performed to improve the accuracy of the calibrated extrinsic parameters.

[0049] In a possible implementation, obtaining the point cloud data of at least one calibration target collected by a lidar on a vehicle and the image frame data of the at least one calibration target collected by a camera on the vehicle includes:

[0050] Obtaining a solution set and a verification set, where the solution set includes the point cloud data corresponding to L1 timestamps and the image frame data corresponding to the L1 timestamps; the verification set includes the point cloud data corresponding to L2 timestamps and the image frame data corresponding to the L2 timestamps;

[0051] After obtaining the first extrinsic parameter and the second extrinsic parameter using the point cloud data corresponding to the i-th timestamp among the L1 timestamps and the image data corresponding to the i-th timestamp, the verification set is used to verify the first extrinsic parameter and the second extrinsic parameter;

[0052] When the calibration condition is not met during verification, continue to adjust the first extrinsic parameter and the second extrinsic parameter using the point cloud data corresponding to the (i + 1)-th timestamp among the L1 timestamps and the image frame data corresponding to the (i + 1)-th timestamp until the calibration condition is met;

[0053] When the calibration condition is met during verification, the first extrinsic parameter that meets the calibration condition is used as the extrinsic parameter of the lidar relative to the vehicle obtained by calibration, and the second extrinsic parameter that meets the calibration condition is used as the extrinsic parameter of the camera relative to the vehicle obtained by calibration.

[0054] In the above implementation, the extrinsic parameters of the lidar and the camera are jointly adjusted by constructing a solution set and a verification set, and the accuracy of calibration is improved by evaluating the credibility of calibration.

[0055] Second aspect, a calibration device is provided. The calibration device can be used to implement the functions described in the method of the first aspect. In an optional implementation, the calibration device includes a processing unit (sometimes also referred to as a processing module) and a transceiver unit (sometimes also referred to as a transceiver module). The transceiver unit can implement the sending function and the receiving function. When the transceiver unit implements the sending function, it can be referred to as a sending unit (sometimes also referred to as a sending module). When the transceiver unit implements the receiving function, it can be referred to as a receiving unit (sometimes also referred to as a receiving module). The sending unit and the receiving unit can be the same functional module, and this functional module is called the transceiver unit, which can implement the sending function and the receiving function; or, the sending unit and the receiving unit can be different functional modules, and the transceiver unit is a general term for these functional modules.

[0056] Third aspect, an embodiment of the present application provides a calibration device, and the device includes a processor and a memory. The memory is used to store program code. The processor is used to read and execute the program code stored in the memory to implement the method described in the first aspect or any design of the first aspect.

[0057] Fourth aspect, an embodiment of the present application further provides a computer storage medium. Software programs are stored in the storage medium, and when the software programs are read and executed by one or more processors, the methods provided by any design of the first aspect can be implemented.

[0058] Fifth aspect, an embodiment of the present application provides a computer program product containing instructions. When it runs on a computer, it causes the computer to execute the method provided by any design of the first aspect above.

[0059] Sixth aspect, an embodiment of the present application provides a chip, and the chip includes a processor. The processor is used to execute the method provided by any design of the first aspect.

[0060] In a possible design, the chip further includes a communication interface, and the communication interface is coupled to the processor.

[0061] In a possible design, the chip is connected to a memory and is used to read and execute the software program stored in the memory to implement the method provided by any design of the first aspect.

[0062] Seventh aspect, an embodiment of the present application provides a calibration device, and the calibration device includes at least one mirror and at least one calibration pattern; the at least one mirror is used to calibrate the lidar on the vehicle; the at least one calibration pattern is used to calibrate the camera on the vehicle.

[0063] In a possible design, the calibration pattern is a checkerboard or a two-dimensional code.

[0064] Based on the implementations provided in the above aspects, this application can be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of a vehicle coordinate system;

[0066] Figure 2 It is a schematic diagram of a radar coordinate system;

[0067] Figure 3 It is a schematic diagram of the geometric relationship between a radar coordinate system and a vehicle coordinate system;

[0068] Figure 4 It is a schematic diagram of a camera coordinate system and an image coordinate system;

[0069] Figure 5 It is a schematic diagram of an application scenario provided by an embodiment of this application;

[0070] Figure 6 It is a schematic diagram of a side view of a calibration target provided by an embodiment of this application;

[0071] Figures 7A - 7E It is a schematic diagram of a calibration target provided by an embodiment of this application;

[0072] Figure 8 It is a schematic diagram of a calibration system architecture provided by an embodiment of this application;

[0073] Figure 9 It is a schematic diagram of a calibration method flow provided by an embodiment of this application;

[0074] Figure 10 It is a schematic diagram of the sensing area of a lidar provided by an embodiment of this application;

[0075] Figure 11 It is a schematic diagram of a process for determining the coordinates of the geometric center of a set geometric shape provided by an embodiment of this application;

[0076] Figure 12 It is a schematic diagram of a convolution operator provided by an embodiment of this application;

[0077] Figure 13 It is a schematic diagram of a process for determining the coordinates of a set pattern provided by an embodiment of this application;

[0078] Figure 14 It is a schematic diagram of an external parameter optimization process provided by an embodiment of this application;

[0079] Figure 15 It is a schematic diagram of the structure of a calibration device provided by an embodiment of this application;

[0080] Figure 16 Schematic structural diagram of another calibration device provided by an embodiment of the present application. Detailed implementation manners

[0081] To make the present application easier to understand, some basic concepts related to the embodiments of the present application are first explained below. It should be noted that these explanations are for the purpose of making the embodiments of the present application easier to understand, and should not be regarded as a limitation on the protection scope required by the present application.

[0082] (1) LiDAR

[0083] There are transmitters and receivers in a LiDAR. The transmitter emits a laser beam. After the laser beam encounters a target (such as an object around a vehicle), it is reflected and returns to the receiver. The distance between the transmitter and the target can be calculated by multiplying the time interval between the transmission time and the reception time by the speed of light and then dividing by 2.

[0084] LiDARs include single-beam laser transmitters, four-line LiDARs, sixteen-line LiDARs, thirty-two-line LiDARs, and so on. Taking a single-beam laser transmitter as an example, the single-beam laser transmitter can rotate at a constant speed inside the LiDAR and emit a laser each time it rotates a small angle. After polling a certain angle, a complete frame of data is generated. Therefore, the data of a single-line LiDAR can be regarded as a row of dot matrices at the same height. The four-line LiDAR polls four laser transmitters. After a polling cycle, a frame of LiDAR point cloud data is obtained. The LiDAR point cloud data can form planar information, and the height information of obstacles can be obtained. Therefore, the more the number of laser transmitters, the higher the efficiency and the richer the information obtained.

[0085] (2) Neural network, which is a branch of machine learning. The neural networks mentioned in the present application can include various types, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), residual networks, attention networks, self-attention networks, neural networks using the transformer model, or other neural networks, etc.

[0086] (3) Vehicle coordinate system.

[0087] Please refer to Figure 1 As shown, it is a schematic diagram of the vehicle coordinate system. The origin of the vehicle coordinate system can be located at any position on the vehicle (such as the center of mass, the ground below the midpoint of the rear axle of the vehicle, or a point on the front axle). The x-axis points forward along the vehicle head, the z-axis is perpendicular to the vehicle chassis and points upward. According to the definition of the right-hand coordinate system, when facing the front of the vehicle, the y-axis points to the left side of the vehicle.

[0088] The vehicle is stationary and parked at a certain position. The vehicle coordinate system can also be understood as the world coordinate system.

[0089] (4) The lidar coordinate system, abbreviated as the radar coordinate system.

[0090] Please refer to Figure 2 As shown, it is a schematic diagram of the radar coordinate system. The origin of the radar coordinate system is at the installation position of the radar on the vehicle. The x-axis, y-axis, and z-axis can have multiple definition methods. For example, they are designed by the lidar manufacturers, and different manufacturers have different designs. Generally speaking, there is a rotation and translation relationship between the radar coordinate system and the vehicle coordinate system.

[0091] Generally, there is a rotation and translation relationship between the radar coordinate system and the vehicle coordinate system. During the driving process of the vehicle, to ensure driving safety, the vehicle needs to know the positions of surrounding objects (such as obstacles in front of the vehicle) in the real environment. That is to say, it is necessary to obtain the position coordinates of the objects around the vehicle in the inertial coordinate system. For this purpose, one solution is that the on-vehicle radar can sense the positions of surrounding objects and represent these positions in the radar coordinate system. Since there is a rotation and translation relationship between the radar coordinate system and the vehicle coordinate system, the coordinates of the target in the radar coordinate system are different from those in the real world. Therefore, the coordinates of the target in the radar coordinate system can be converted to the vehicle coordinate system to determine the position of the target in the real environment. It should be understood that during the process of converting one coordinate system to another, the relative position relationship (such as rotation and translation relationship) between these two coordinate systems needs to be used. Therefore, to achieve the conversion from the radar coordinate system to the vehicle coordinate system, the relative position relationship between the radar coordinate system and the vehicle coordinate system needs to be used, and this relative position relationship is used for the coordinate conversion from the radar coordinate system to the vehicle coordinate system. Among them, the parameters required for the conversion from the radar coordinate system to the vehicle coordinate system are called the external parameters of the radar (abbreviated as external parameters).

[0092] (5) The external parameters of the on-vehicle radar (abbreviated as external parameters)

[0093] As described above, there is a relative position relationship between the radar coordinate system and the vehicle coordinate system, and the relative position relationship includes rotation and translation relationships. The relative position relationship is called the external parameters of the on-vehicle radar. That is to say, the external parameters of the on-vehicle radar include the rotation and translation relationships of the radar coordinate system of the on-vehicle radar relative to the vehicle coordinate system.

[0094] Please refer to Figure 3 As shown, it is a schematic diagram of the relative position relationship between the radar coordinate system and the vehicle coordinate system. As Figure 3 shown, the origin of the radar coordinate system is defined as O L , and its coordinate system is O L -X L YL Z L The origin of the vehicle coordinate system is defined as O V and its coordinate system is O V -X V Y V Z V The external parameters of the vehicle-mounted radar include the rotation and translation relationships of the radar coordinate system relative to the vehicle coordinate system. Among them, the rotation relationship is described by the rotation angle, and the translation relationship is described by the translation distance.

[0095] The rotation angle of the radar coordinate system relative to the vehicle coordinate system can be described by three attitude angles, namely, the pitch angle β (pitch), the yaw angle γ (yaw), and the roll angle α (roll). Among them, the pitch angle β (pitch) refers to the angle of counterclockwise rotation around the Y L axis; the yaw angle γ (yaw) refers to the angle of counterclockwise rotation around the Z L axis; the roll angle α (roll) refers to the angle of counterclockwise rotation around the X L axis. In other words, after the radar coordinate system O L -X L Y L Z L rotates -γ around the Z L axis, -β around the Y L axis, and then -α around the X L axis, its coordinate axes have the same directions as the three axes in the vehicle coordinate axes O V -X V Y V Z V Among them, -γ refers to the direction opposite to γ. Similarly, -β is opposite to β, and -α is opposite to α.

[0096] The translation distance of the radar coordinate system relative to the vehicle coordinate system can be described by three translation distances, namely, Δx, Δy, and Δz. Among them, Δx is the projection value of the distance from the origin O L of the radar coordinate system to the origin O V of the vehicle coordinate system on the x-axis. Δy is the projection value of the distance from the origin O L of the radar coordinate system to the origin O V of the vehicle coordinate system on the y-axis. Δz is the projection value of the distance from the origin O L of the radar coordinate system to the origin O V of the vehicle coordinate system on the z-axis. That is to say, after translating the origin O L of the radar coordinate system by -Δx on the x-axis, -Δy on the y-axis, and -Δz on the z-axis, the origin O Land the vehicle coordinate system. Here, -Δx means the direction opposite to the Δx direction, -Δy is opposite to the Δy direction, and -Δz is opposite to the Δz direction.

[0097] Therefore, after determining the rotation angle (including three attitude angles) and translation distance of the radar coordinate system relative to the vehicle coordinate system, the coordinates of the target (such as an object around the vehicle) in the radar coordinate system can be converted to the vehicle coordinate system using the rotation angle and translation distance. Since the rotation angle and translation distance are collectively referred to as the external parameters of the vehicle-mounted radar, the process of determining the rotation angle and translation distance of the radar coordinate system relative to the vehicle coordinate system can be called the calibration process of the external parameters of the vehicle-mounted radar, and the calibration can be understood as determination, acquisition, calculation, etc.

[0098] The relationship between the radar coordinate system and the world coordinate system can be described by the rotation matrix R’ and the translation matrix T’.

[0099]

[0100] The relationship between the radar coordinate system and the world coordinate system can be described by the following formula (1).

[0101]

[0102] where, x v , y v and z v represent the coordinates of the point cloud in the world coordinate system. x l , y l and z l represent the coordinates of the point cloud in the lidar coordinate system.

[0103] (6) Camera coordinate system.

[0104] Camera extrinsic parameters: The rotation and translation transformations of the camera in the pinhole camera model relative to a certain coordinate system (such as the world coordinate system or the pose of a certain reference camera), that is, the 6 degrees of freedom (6dof) pose of the camera in this coordinate system. It represents the translational motion along three directions and the rotational motion around three axes.

[0105] A specific example of an image coordinate system can be seen in Figure 4 as shown. Point O i is the intersection of the camera optical axis and the camera imaging plane, and it is the origin of the image physical coordinate system. (u, v) represents the column number and row number of the pixel. Among them, (O p , u, v) constitutes the pixel plane coordinate system. The origin O pLocated at the upper left corner of the camera imaging plane, two coordinate axes (O p u-axis and O p v-axis) point to the right and downward respectively. The origin O i of the image physical coordinate system is located at the center of the pixel plane coordinate system, with coordinates (u0, v0). Two coordinate axes (O i x-axis and O i y-axis) point to the right and downward respectively. Let dx and dy represent the physical sizes of a pixel along the u-axis and v-axis directions respectively. Then the relationship between the pixel plane coordinate system and the image coordinate system is shown in Equation (2).

[0106]

[0107] The above Equation (2) can be further expressed as Equation (3):

[0108]

[0109] A specific example of a camera coordinate system can be seen in Figure 4 as shown. The camera coordinate system is a spatial coordinate system, and its origin O c is located at the camera optical center. The O i point is the intersection of the camera optical axis and the camera imaging plane, that is, the origin of the image physical coordinate system. As Figure 4 shown, the O c x c -axis and O c y c -axis of the camera coordinate system are respectively parallel to the O i x-axis and O i y-axis of the image coordinate system. The O c z c -axis passes through the point O i . The distance from the point O i to the point O c is the focal length, denoted by f. According to the imaging projection relationship, it can be known that the camera coordinate system and the image coordinate system satisfy the relationship shown in the following Equation (4):

[0110]

[0111] Generally, there is a rotational and translational relationship between the camera coordinate system and the vehicle coordinate system. The relationship between the camera coordinate system and the world coordinate system can be described by the rotation matrix R and the translation parameter T. Thus, the homogeneous coordinates of a point P in the world coordinate system and the camera coordinate system are respectively

[0112] (x w , y w , z w ) and (x c , y c , zc ) satisfies the relationship shown in the following formula (5).

[0113] Wherein, R is a 3×3 rotation matrix, and T is a 3×1 translation parameter.

[0114] Based on the relationships among the above pixel plane coordinate system, image coordinate system, camera coordinate system, and world coordinate system, the relationship shown in formula (5).

[0115]

[0116] Wherein, in formula (6), is the internal parameter matrix, on the right side of the equal sign is the external parameter matrix.

[0117] dx and dy represent how many length units a pixel in the x direction and the y direction respectively occupy, that is, the size of the actual physical value represented by a pixel, which is the key to realizing the conversion between the camera coordinate system and the image coordinate system.

[0118] In the description of the present application, unless otherwise specified, "a plurality of" means two or more than two. In addition, " / " means that the associated objects before and after are in an "or" relationship. For example, A / B can represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Among them, A and B can be singular or plural. And, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the terms such as "first" and "second" do not limit the quantity and execution order, and the terms such as "first" and "second" do not necessarily limit to be different. It should also be noted that, unless otherwise specified, the specific description of some technical features in one embodiment can also be applied to explain the corresponding technical features mentioned in other embodiments.

[0119] As can be seen from the background art, in the current calibration method, the calibration process is relatively complex and the accuracy is low. The embodiments of the present application provide a calibration method and device, which do not need to separately calibrate the geometric relationship between the camera and the lidar and the vehicle, simplify the calibration process, and improve the calibration accuracy.

[0120] See Figure 5As shown in the figure, it is a schematic diagram of an application scenario provided by an embodiment of the present application. The vehicle to be calibrated is parked statically in the calibration site. One or more calibration targets are deployed in the calibration scenario. The calibration target can also be called a calibration device or a calibration body or a calibration plate, etc. The calibration target is used for the joint calibration of the camera and the lidar. The calibration target is deployed around the vehicle and within the viewing angles of the camera and the lidar on the vehicle.

[0121] In some embodiments, before performing the calibration, a dotting tool can be used to obtain the deployment position of the calibration target and the position of the vehicle. In the embodiments of the present application, the device form and dotting method of the dotting tool are not specifically limited.

[0122] The calibration target in the embodiments of the present application includes one or more first-type calibration objects for calibrating the lidar, and one or more second-type calibration objects for calibrating the camera.

[0123] In a possible embodiment, the calibration area on the first-type calibration object can be a set geometric shape, such as a circle, a square, or a triangle, etc., or other irregular shapes can also be used.

[0124] In one example, the first-type calibration object can be a mirror. The shape of the mirror can be a regular shape such as a circle, a square, or a triangle, or an irregular shape can also be used. The mirror can be attached to the calibration target. For example, if the calibration target is a flat plate, the mirror can be attached to the flat plate. Another example is that the mirror can be attached to the wall. The light emitted by the lidar hits the mirror, and due to the reflection principle of the mirror, hole-shaped edge points will be formed on the calibration device surface. The edge points can be understood as the points where the point cloud changes violently at the edge of the mirror when the light hits the mirror, thus forming holes.

[0125] In another example, the first-type calibration object can be a calibration plate with holes. The shape of the holes can be a regular shape such as a circle, a square, or a triangle, or an irregular shape can also be used. The holes dug on the calibration plate are columnar structures. In some possible scenarios, in order to prevent the light rays emitted by the lidar from hitting the inner wall of the holes, the inner wall of the dug holes can be chamfered. See Figure 6 As shown in the figure, taking the hole as a circular hole as an example, the side view of the calibration target is as shown in Figure 6 As shown in the figure. The chamfering angle can be 30 degrees, 60 degrees, or other angles can also be used. For example, the chamfering angle can be set according to the distance between the lidar and the calibration target, as long as the angles at which the light rays can pass through the holes are applicable to the present application. The light rays of the lidar can pass through the holes completely, thus forming edge points around the holes. The edge points can be understood as the points where the point cloud changes violently at the position of the dug holes, thus forming holes.

[0126] In another possible embodiment, the calibration area on the second type of calibration object may be a set calibration pattern. The calibration pattern may be composed of lines and points and can be any design that can easily detect regular shapes such as corner points, lines, and circles. For example, the calibration pattern may be a QR code, a checkerboard, a combination of several regular patterns (custom pattern), etc. In addition, the method of setting the calibration object on the calibration target is not limited either. For example, it can be set on the calibration target by means of a metal plate, a wooden board, a plastic plate, a cardboard, a cloth, glass, ceramics, quartz, or an easily pasted printed matter, paint printing, pigment spraying, etc. By selecting different types and different setting methods to set the calibration object on the calibration target, not only is the setting of the calibration object more flexible, but also the popularization of this method is increased.

[0127] For example, refer to Figure 7A As shown, it is a schematic diagram of a possible calibration target. Figure 7A In it, take the calibration target including multiple circular reflectors and multiple checkerboard patterns as an example. Figure 7A Take the number of circular reflectors and checkerboard patterns both being 3 as an example. Figure 7A In it, take the calibration target being a flat plate as an example.

[0128] For another example, refer to Figure 7B As shown, take the circular reflectors and checkerboard patterns being attached to the wall surface of the space where the vehicle is located as an example.

[0129] For yet another example, refer to Figure 7C As shown, it is a schematic diagram of another possible calibration target. Figure 7C In it, take the calibration target including multiple circular holes and multiple checkerboard patterns as an example. Figure 7C Take the number of circular reflectors and checkerboard patterns both being 5 as an example. The calibration target can be a flat plate or a wall surface. For example, if the calibration target is a wall surface, holes can be dug on the wall surface.

[0130] For yet another example, refer to Figure 7D As shown, it is a schematic diagram of yet another possible calibration target. Figure 7D In it, take the calibration target including multiple square mirrors and multiple checkerboard patterns as an example. Figure 7D Take the number of square mirrors and checkerboard patterns both being 2 as an example. The calibration target can be a flat plate or a wall surface. For example, if the calibration target is a wall surface, holes can be dug on the wall surface.

[0131] For yet another example, refer to Figure 7E As shown, it is a schematic diagram of yet another possible calibration target. [[ID=�0]] Figure 7E In it, take the calibration target including multiple triangular holes and multiple QR code patterns as an example. Figure 7E Take the number of triangular holes and QR code patterns both being 2 as an example. The calibration target can be a flat plate or a wall surface. For example, if the calibration target is a wall surface, holes can be dug on the wall surface.

[0132] The calibration target provided by the embodiment of the present application is simple to manufacture. When a flat panel is used as the calibration target, it is convenient to move. When a wall surface is used as the calibration target, the existing buildings in the calibration scene can be utilized, which can save costs and improve space utilization.

[0133] The above Figures 7A - 7E is only taken as several examples and does not limit the specific form of the calibration target.

[0134] See Figure 8 As shown, it is a schematic diagram of a calibration system architecture provided by the embodiment of the present application. The calibration system includes a vehicle and one or more calibration targets. One or more lidars to be calibrated and one or more cameras to be calibrated are included on the vehicle. The camera provides visual information, and the lidar provides distance and three-dimensional information. Cameras and lidars are deployed on the vehicle for data fusion to obtain the perception of the vehicle's surrounding environment. A calibration device, also called a calibration module, can also be deployed on the vehicle, and other names can also be used. The embodiment of the present application does not limit this. In subsequent descriptions, the calibration device is taken as an example, and the calibration device is coupled with the camera and the lidar. In some possible application scenarios, the calibration device can also be deployed outside the vehicle and establish a communication connection with the camera and the lidar. For example, through a network, the network can be a wide area network or a local area network, or a combination of the two. In some embodiments, when the calibration device is deployed outside the vehicle, the calibration device can run in a cloud computing device system (which can include at least one cloud computing device, such as a server, etc.), can also run on a physical server, can also run in an edge computing device system (which can include at least one edge computing device, such as a server, a desktop computer, etc.), and can also run on various terminal computing devices, such as a laptop computer, a personal desktop computer, a mobile phone, a tablet computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and so on. In some embodiments, a storage unit ( Figure 8 not shown in the figure) can also be deployed in the calibration system. The storage unit can be independent of the calibration device or can be deployed inside the calibration device. For example, the storage unit can be a cloud storage server or a physical storage server. After the lidar and the camera collect data, the data can be sent to the storage unit for storage. The calibration device can obtain data from the storage unit. The storage unit has an independent data matching and transmission interface for transmitting data to the calibration device. The calibration device can have a data receiving interface for receiving point cloud data and image frame data.

[0135] See Figure 9As shown, it is a schematic flowchart of a calibration method provided by an embodiment of the present application. This calibration method can be executed by a calibration device. The calibration device calibrates the lidar and camera on the vehicle.

[0136] S901, the calibration device acquires the point cloud data of the calibration target collected by the lidar and the image frame data of the calibration target collected by the camera. The calibration target includes at least one first type of calibration object and at least one second type of calibration object.

[0137] One or more calibration targets are deployed inside the sensing area (or visible range) of the lidar. Refer to Figure 10 As shown, the lidar deployed on the vehicle can sense the fan-shaped area shown by the solid line frame. This fan-shaped area can be understood as the sensing area of the lidar. When the lidar senses that there is a calibration target in the sensing area, it transmits a signal (such as point cloud data) to the calibration device.

[0138] The above one or more calibration targets also exist in the visible range of the camera. The visible range of the camera can also be understood as a fan-shaped area. After the camera acquires the image frame data, it transmits the image frame data to the calibration device.

[0139] In some embodiments, the point cloud data acquired by the calibration device can include multiple groups or multiple frames, which can be continuously collected by the lidar and sent to the calibration device. The timestamps corresponding to different frames of point cloud data are different. In other embodiments, the point cloud data acquired by the calibration device includes multiple groups or multiple frames, which can be obtained by the lidar continuously collecting multiple frames of point cloud data and then performing frame extraction and sending the multiple frames of point cloud data to the calibration device. In still other embodiments, the frame extraction process for the point cloud data can also be executed by the calibration device.

[0140] In some embodiments, the image frame data acquired by the calibration device can include multiple frames, which can be continuously collected by the camera and sent to the calibration device. The timestamps corresponding to different frames of image frame data are different. In other embodiments, the image frame data acquired by the calibration device includes multiple frames, which can be obtained by the camera continuously collecting multiple frames of image frame data and then performing frame extraction and sending it to the calibration device. In still other embodiments, the frame extraction process can also be executed by the calibration device. In still other embodiments, the frame extraction process for the image frame data can also be executed by the calibration device.

[0141] After obtaining the point cloud data and image frame data, the calibration device can also preprocess the point cloud data and image frame data. For example, it can determine whether there are abnormalities in the read data. For example, when the camera is abnormal, a complete image frame data is not obtained, or the camera is abnormal and frames are lost, or the image content of the transmitted image frame is missing. For another example, when the lidar is abnormal, point cloud data cannot be scanned and obtained. In some embodiments, the lidar and the camera can collect data synchronously. When it is determined that the point cloud data and image frame data of the current frame are normal, the time stamps of the point cloud data and image frame data can be matched for subsequent processing.

[0142] In a possible implementation manner of the present application, when the calibration device determines the matched point cloud data and image frame data, it can send the matched point cloud data and image frame data to the thread queue in the calibration device, and read the data from the thread queue during the subsequent calibration process.

[0143] S902. Determine the position coordinates of at least one first type of calibration object in the lidar coordinate system according to the point cloud data, and determine the position coordinates of at least one second type of calibration object in the image coordinate system of the camera according to the image frame data.

[0144] Exemplarily, there is a calibration area on the first type of calibration object. As described above, the calibration area can be a set geometric shape. The position coordinates of the first type of calibration object in the lidar coordinate system can be the coordinates of the geometric center of the set geometric shape. For example, if the set geometric shape is a circle, the position coordinates of the first type of calibration object in the lidar coordinate system can be the coordinates of the center of the circle in the lidar coordinate system. For example, if the set geometric shape is a square or a triangle, the position coordinates of the first type of calibration object in the lidar coordinate system can be the coordinates of the geometric center of the square or triangle in the lidar coordinate system.

[0145] Exemplarily, there is a calibration area on the second type of calibration object. As described above, the calibration area on the second type of calibration object is a set calibration pattern. The calibration pattern can be composed of lines and points. The position coordinates of the second type of calibration object in the image coordinate system of the camera can be the coordinates of the calibration points and / or calibration lines of the second type of calibration object in the image coordinate system of the camera. For example, if the calibration pattern is a checkerboard or a QR code, the calibration points can be the corner points of the checkerboard, and the calibration lines can be the lines on the upper edge of the checkerboard or the lines on the edge of the QR code. Of course, the calibration points and calibration lines can be set on the second type of calibration object, and the specific setting positions can be set according to requirements.

[0146] The determination method of S902 will be described in detail later, and will not be elaborated here.

[0147] S903. Calibrate the first extrinsic parameter of the lidar relative to the vehicle based on the position coordinates of at least one first-type calibration object in the world coordinate system and the position coordinates of at least one first-type calibration object in the lidar coordinate system, and calibrate the second extrinsic parameter of the camera relative to the vehicle based on the position coordinates of at least one second-type calibration object in the world coordinate system and the coordinates of at least one second-type calibration object in the image coordinate system of the camera.

[0148] Among them, the world coordinate system is established with the vehicle as a reference, and the world coordinate system can also be understood as the vehicle coordinate system. The position coordinates of at least one first-type calibration object in the world coordinate system can be measured by a dotting tool. The position coordinates of at least one second-type calibration object in the world coordinate system can also be measured by a dotting tool.

[0149] The process of determining the three-dimensional coordinates of the geometric center of the set geometric shape in the first-type calibration object in the lidar coordinate system in the embodiments of the present application is described exemplarily as follows. Refer to Figure 11 As shown, determining the three-dimensional coordinates of the geometric center of the set geometric shape in the lidar coordinate system may include: A1 target detection, A2 geometric segmentation, A3 edge clustering, A4 fitting geometric center.

[0150] A1 Target detection:

[0151] The calibration device determines the point cloud data of the region of interest (ROI) from the point cloud data. The region of interest includes the calibration object of the first type. Then, the edge points with significant features in the point cloud data of the region of interest are determined. Having significant features can be understood as the region where the point cloud changes violently in the region of interest, or the region with obvious changes or discontinuities.

[0152] In a possible implementation, the edge points with significant features in the point cloud data of the region of interest can be determined according to the three-dimensional coordinates, laser scan angle, and ring ID of each point cloud in the point cloud data. The ring ID represents the arrangement serial number of the point cloud.

[0153] Exemplarily, the calibration device sorts the point cloud within the ROI using the ring ID and the scanning angle. For example, the sorting criterion is: from top to bottom, from left to right. The sorting method is only an example, and other methods can also be adopted. The embodiments of the present application do not limit this. Then, traverse this part of the point cloud in order, and project this part of the point cloud onto the fitted target plane according to the laser emission direction. Further, loop through the sorted order to determine whether a point cloud is the starting or ending position of a row. If so, skip it; if not, determine whether the y-axis projection distance between this point cloud and the next point cloud is greater than the first set threshold. If the y-axis projection distance between this point cloud and the next point cloud is greater than the first set threshold, then this point cloud is an edge point with significant features. Also, loop through the sorted order to determine whether a point cloud is the starting or ending position of a column. If so, skip it (that is, this point cloud is not an edge point with significant features); if not, determine whether the z-axis projection distance between this point cloud and the next point cloud is greater than the second set threshold. If the y-axis projection distance between this point cloud and the next point cloud is greater than the second set threshold, then this point cloud is an edge point with significant features.

[0154] A2 Geometric segmentation:

[0155] The calibration device maps the edge points with significant features into the two-dimensional image coordinate system to obtain a two-dimensional image. Among them, the three-dimensional coordinate features of the edge points are retained. Then, perform template segmentation on the two-dimensional image to obtain at least one light cluster region, and the at least one light cluster region corresponds to the at least one first type of calibration object one by one; the light cluster region is used to indicate the interval range of the set geometric shape on the corresponding first type of calibration object in the two-dimensional image.

[0156] For example, traverse the two-dimensional image using a convolution kernel operator, that is, obtain the convolved image by means of template segmentation, that is, obtain each light cluster region. As an example, the form of the convolution kernel operator can be as Figure 12 shown. Figure 12 In, take the pixel value of a part of the area in the middle layer as -1, the pixel value of the outermost layer area as 0, and the pixel value of the interlayer between the outermost layer and the middle layer as 1 as an example. Figure 11 This is only an example of the convolution kernel operator, and the embodiments of the present application do not specifically limit the form of the convolution kernel operator.

[0157] A3 Edge clustering:

[0158] The calibration device performs maximization suppression on each of the at least one light cluster region, and processes the at least one light cluster region after maximization suppression to obtain the edge points of the set geometric shape on the at least one first type of calibration object. By performing maximization suppression on each of the at least one light cluster region respectively, the edge points of the geometric shape can be initially obtained. Outliers among the initially obtained edge points of the geometric shape can also be further filtered out through a clustering algorithm, thereby obtaining the edge points of the geometric shape. Maximization suppression, also known as maximum value suppression, is a signal processing technique used to suppress the maximum values in a signal to reduce noise signals and retain meaningful signals.

[0159] Exemplarily, if the set geometric shape is a circular hole, after each light cluster region undergoes maximization suppression, the edge points of each circular hole can be obtained by using the three-dimensional coordinates of the edge points of each light cluster region, and then the outliers among the edge points of the circular hole can be filtered out through a clustering algorithm, leaving the edge points of the circular hole to be fitted.

[0160] A4 Fit the geometric center.

[0161] The calibration device determines the three-dimensional coordinates of the geometric midpoint of the set geometric shape on the at least one first type of calibration object respectively according to the three-dimensional coordinates of the edge points of the set geometric shape on the at least one first type of calibration object. For example, for each set geometric shape, a fitting operation can be performed on the edge points of each set geometric shape to obtain the three-dimensional coordinates of the geometric center of each geometric shape in the laser coordinate system. For example, the methods used in the fitting operation can include but are not limited to the random sample consensus (ransac) algorithm, the least squares method, the mean method, etc.

[0162] In some possible implementation manners, after fitting the geometric center, A5 geometric center registration can also be performed. The calibration device pairs the three-dimensional coordinates of the geometric center of the geometric shape on the first type of calibration object in the world coordinate system with the three-dimensional coordinates of the geometric center of the geometric shape on the first type of calibration object in the laser coordinate system one by one. For example, the pairing operation can be performed through a global matching algorithm. The global matching algorithm can include but is not limited to the ransac algorithm, the iterative closest point (ICP) algorithm, the singular value decomposition algorithm, etc.

[0163] The process of determining the position coordinates of at least one second-type calibration object in the image coordinate system of the camera in the embodiments of the present application is described exemplarily as follows. Taking the set pattern on the second-type calibration object as an example. First, the edge detection algorithm can be used to detect the image data of the set pattern from the image frame data; then, at least one calibration point and / or at least one calibration line in the set pattern are detected according to the image data of the second-type calibration object. As an example, taking the calibration point as a corner point and the calibration line as at least one edge line. Refer to Figure 13 As shown, determining the position coordinates of the calibration point (taking the corner point as an example) and / or the calibration line in the image coordinate system of the camera may include: B1 image preprocessing, B2 edge detection, B3 corner detection, B4 feature description, B5 threshold screening, and B6 feature matching.

[0164] B1 Image preprocessing:

[0165] The calibration device performs preprocessing on the acquired image frame data. For example, the preprocessing may include, but is not limited to: grayscale conversion, denoising, and image enhancement, etc. Through preprocessing, the visibility of feature points can be enhanced and the influence of noise can be reduced. This image preprocessing is an optional step. In some possible scenarios, this step may not be executed.

[0166] B2 Edge detection:

[0167] The calibration device can use the edge detection algorithm according to the image frame data to detect the image data of the set pattern on the calibration target. Taking the set pattern as a two-dimensional code or a checkerboard as an example. The calibration device detects the two-dimensional code or checkerboard image on the calibration target through the edge detection algorithm. Exemplarily, the edge detection algorithm may include, but is not limited to: Canny edge detection, Sobel operator, etc.

[0168] B3 Corner detection:

[0169] Taking the set pattern as a two-dimensional code or a checkerboard as an example, the calibration device can perform corner detection on the image data of the set pattern (two-dimensional code or checkerboard) to obtain multiple corner points in the set pattern. Through B2 edge detection, multiple edge lines can also be obtained.

[0170] Exemplarily, the corner detection can adopt one or more of the following algorithms: Harris corner detection, Shi-Tomasi corner detection, etc.

[0171] B4 Feature description:

[0172] The calibration device determines the descriptors of the multiple corner points and / or the descriptors of the multiple edge lines. For example, through a feature description algorithm, the descriptors of the multiple corner points and / or the descriptors of the multiple edge lines are calculated. The feature description algorithm can adopt one or more of the following: Scale Invariant Feature Transformation (SIFT), Speeded Up Robust Features (SURF), and Oriented FAST and Rotated BRIEF (ORB), etc. BRIEF is Binary Robust Independent Elementary Features.

[0173] B5 Threshold Screening:

[0174] The calibration device screens out at least one corner point as a calibration point from the multiple corner points according to the descriptors of the multiple edge points, and / or screens out at least one edge line as a calibration line from the multiple edge lines, so as to obtain the coordinates of at least one corner point as a calibration point and / or at least one edge line as a calibration line in the image coordinate system of the camera.

[0175] Exemplarily, during the above screening, at least one corner point as a calibration point can be screened out from the multiple corner points, and at least one edge line as a calibration line can be screened out from the multiple edge lines by means of threshold screening.

[0176] The above B3 corner detection or B2 edge detection can obtain the response values of points and lines. The response value reflects the gray-scale change situation in the area around the pixel point or line. A larger response value means that there are obvious corner points in the area around the pixel point and obvious feature line segments around the line segment. A lower threshold will detect more corner points, including corner points with lower intensity, while a higher threshold will filter out some weaker corner points. Furthermore, during the B5 threshold screening process, by using the intensity threshold, appropriate corner points and edge lines can be selected for subsequent processing according to the requirements of the actual application and the characteristics of the image. It can be understood that if the threshold is set too low, it may lead to detecting too many noise points or weaker corner points; if the threshold is set too high, some weak corner points or correct corner points may be missed. Thus, a more appropriate intensity range can be set according to the requirements of the actual application.

[0177] B6 Feature Matching:

[0178] In the case where there are multiple calibration objects of the second type on the calibration target, a feature point matching algorithm can be used to match the corner points and / or edge lines in the images of different preset patterns with the three-dimensional coordinates of the calibration points and / or calibration lines in the world coordinate system one by one.

[0179] The following is an exemplary description of the method for calibrating the first external parameter of the lidar relative to the vehicle according to the position coordinates of at least one calibration object of the first type in the world coordinate system and the position coordinates of at least one calibration object of the first type in the lidar coordinate system, and the method for calibrating the second external parameter of the camera relative to the vehicle according to the position coordinates of the at least one calibration object of the second type in the world coordinate system and the coordinates of the at least one calibration object of the second type in the image coordinate system of the camera.

[0180] In some embodiments, linear solution can be used to calibrate the first external parameter and the second external parameter.

[0181] For example, the number of calibration objects of the first type on the calibration target can be N, where N is an integer greater than 1. Then, based on the three-dimensional coordinates of the geometric centers of the N calibration objects of the first type in the world coordinate system and the three-dimensional coordinates of the geometric centers of the N calibration objects of the first type in the lidar coordinate system, the first mapping equation can be solved to obtain the first external parameter of the lidar relative to the vehicle; wherein, the first mapping equation is established based on the mapping relationship between the world coordinate system and the lidar coordinate system. For example, the mapping equation can be constructed based on the above formula (1).

[0182] Also, for example, multiple calibration targets can be used. If the number of calibration objects of the first type included in the multiple calibration targets is N, then, based on the three-dimensional coordinates of the geometric centers of the N calibration objects of the first type in the world coordinate system and the three-dimensional coordinates of the geometric centers of the N calibration objects of the first type in the lidar coordinate system, the first mapping equation can be solved to obtain the first external parameter of the lidar relative to the vehicle; wherein, the first mapping equation is established based on the mapping relationship between the world coordinate system and the lidar coordinate system. For example, the mapping equation can be constructed based on the above formula (1).

[0183] Also, for example, based on the three-dimensional coordinates of the geometric centers of the N calibration objects of the first type in the world coordinate system and the three-dimensional coordinates of the geometric centers of the N calibration objects of the first type in the lidar coordinate system, the first mapping equation can be solved to obtain the first external parameter of the lidar relative to the vehicle; wherein, the first mapping equation is established based on the mapping relationship between the world coordinate system and the lidar coordinate system. For example, the mapping equation can be constructed based on the above formula (1).

[0184] In a possible implementation, after obtaining the above-mentioned first extrinsic parameter, the first extrinsic parameter can be further optimized. For example, the first extrinsic parameter can be optimized by establishing a first global cost function. After determining the first extrinsic parameter, based on the three-dimensional coordinates of the geometric centers of the N first-type calibration objects in the world coordinate system respectively, and the three-dimensional coordinates of the geometric centers of the N first-type calibration objects in the lidar coordinate system respectively, and using the first global cost function to optimize the first extrinsic parameter to obtain the optimized first extrinsic parameter.

[0185] The first global cost function is used to characterize the error between the three-dimensional coordinates of the geometric center of the first-type calibration object in the world coordinate system and the three-dimensional coordinates of the geometric center of the first-type calibration object in the lidar coordinate system transformed to the world coordinate system based on the first extrinsic parameter. For example, the first global cost function can adopt formula (7).

[0186]

[0187] Loss1 represents the loss value. w i represents the coordinates of the input point cloud in the world coordinate system. v i represents the coordinates of the point cloud in the laser coordinate system transformed to the world coordinate system through the first extrinsic parameter. Through the first global cost function, starting from the first parameter, a rough search with a large angle is performed to obtain a rough solution, and then non-linear iteration is performed using multiple rough solutions with smaller loss values, so as to obtain the optimal solution within the full range to obtain the optimized first parameter.

[0188] In another possible implementation, after obtaining the above-mentioned second extrinsic parameter, the second extrinsic parameter can be further optimized. For example, the second extrinsic parameter can be optimized by establishing a second global cost function. After determining the second extrinsic parameter, based on the three-dimensional coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second-type calibration objects in the world coordinate system respectively, and the three-dimensional coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second-type calibration objects in the lidar coordinate system respectively, and using the second global cost function to optimize the second extrinsic parameter to obtain the optimized second extrinsic parameter.

[0189] The second global cost function is used to characterize the error between the coordinates of the calibration mark in the image coordinate system of the camera and the coordinates of the calibration mark in the world coordinate system transformed to the image coordinate system using the second extrinsic parameter, where the calibration mark includes at least one calibration point and / or at least one calibration line. For example, the second global cost function can adopt formula (8).

[0190]

[0191] Loss2 represents the loss value. det i represents the coordinates of the calibration points or calibration lines detected from the image. p i represents the coordinates of the calibration points or calibration lines in the world coordinate system converted to the coordinates in the image coordinate system. Through the second global cost function, starting from the second parameter, a rough search with a large angle is performed to obtain a rough solution, and then multiple rough solutions with smaller loss values are used for non-linear iteration to obtain the optimal solution within the full range, so as to obtain the optimized second parameter.

[0192] In a possible implementation manner of the embodiment of the present application, after obtaining the first parameter (or the optimized first parameter) and the second parameter (or the optimized second parameter), further optimization and adjustment can be performed on the optimized first parameter and the second parameter. In the embodiment of the present application, after obtaining multiple frames of point cloud data of at least one calibration target collected by the lidar on the vehicle and multiple frames of image frame data of the at least one calibration target collected by the camera on the vehicle, a solution set and a verification set can be constructed. The solution set includes the point cloud data corresponding to L1 timestamps and the image frame data corresponding to the L1 timestamps; the verification set includes the point cloud data corresponding to L2 timestamps and the image frame data corresponding to the L2 timestamps.

[0193] Further, the first external parameter and the second external parameter are further optimized by using the solution set and the verification set. See Figure 14 as shown.

[0194] S1401, obtain the first external parameter and the second external parameter by using the point cloud data corresponding to the i-th timestamp among the L1 timestamps in the solution set and the image data corresponding to the i-th timestamp. The solution set is used to accumulate the results of multiple frames of external parameters. At this time, the camera and the lidar have obtained the single-frame optimal value through the external parameter solution, and the accumulated external parameter results are inferred and verified in the verification set. For example, after obtaining the first external parameter and the second external parameter by using the point cloud data corresponding to the i-th timestamp among the L1 timestamps in the solution set and the image data corresponding to the i-th timestamp, the external parameter results of i frames have been accumulated.

[0195] S1402, use the verification set to check the first external parameter and the second external parameter to obtain a check result.

[0196] S1403, determine whether the check result meets the calibration condition. If not, execute S1404; if so, execute S1405.

[0197] S1404. Let \(i = i + 1\) and execute S1401. Continue to adjust the first extrinsic parameter and the second extrinsic parameter using the point cloud data corresponding to the \((i + 1)\)-th timestamp among the \(L1\) timestamps and the image frame data corresponding to the \((i + 1)\)-th timestamp until the calibration meets the calibration conditions.

[0198] S1405. Take the first extrinsic parameter that meets the calibration conditions as the extrinsic parameter of the lidar relative to the vehicle obtained by calibration, and take the second extrinsic parameter that meets the calibration conditions as the extrinsic parameter of the camera relative to the vehicle obtained by calibration.

[0199] By the above method of the solution set + verification set, the credibility of the calibrated first extrinsic parameter and second extrinsic parameter can be improved.

[0200] The following introduces the device for implementing the above method in the embodiments of the present application with reference to the drawings. Therefore, the content above can be used in subsequent embodiments, and the repeated content will not be elaborated.

[0201] Figure 15 It is a structural block diagram of a calibration device 1500 provided in an embodiment of the present application. The calibration device 1500 includes: an acquisition unit 1501 and a processing unit 1502. Among them, the acquisition unit 1501 is used to acquire the point cloud data of at least one calibration target collected by a lidar on the vehicle and the image frame data of the at least one calibration target collected by a camera on the vehicle; each calibration target in the at least one calibration target includes at least one first type of calibration object and at least one second type of calibration object. The processing unit 1502 is used to determine the coordinates of the at least one first type of calibration object in the lidar coordinate system according to the point cloud data, and determine the coordinates of the at least one second type of calibration object in the image coordinate system of the camera according to the image frame data; calibrate the first extrinsic parameter of the lidar relative to the vehicle according to the position coordinates of the at least one first type of calibration object in the world coordinate system and the position coordinates of the at least one first type of calibration object in the lidar coordinate system, and calibrate the second extrinsic parameter of the camera relative to the vehicle according to the position coordinates of the at least one second type of calibration object in the world coordinate system and the position coordinates of the at least one second type of calibration object in the camera's image coordinate system; wherein, the world coordinate system is established with the vehicle as a reference.

[0202] Exemplarily, the acquisition unit 1501 and the processing unit 1502 can be processors, such as an application processor or a baseband processor, and the processor may include one or more central processing modules (central processing unit, CPU).

[0203] Exemplarily, the calibration device 1500 may be a vehicle or an in-vehicle device, or a processing module or a chip system in the vehicle or in-vehicle device, such as an in-vehicle processor or an electronic control unit (ECU).

[0204] Exemplarily, the calibration device 1500 may also be a processing module (such as a processor) within an in-vehicle radar.

[0205] Optionally, the acquisition unit of the calibration device 1500 may also include a receiving unit and a sending unit. The sending unit may be a functional module for performing a sending operation; the receiving unit may be a functional module for performing a receiving operation.

[0206] The division of modules in the embodiments of the present application is illustrative, merely a logical function division. In actual implementation, there may be other division methods. Additionally, in each embodiment of the present application, each functional module may be integrated in a processor, may exist physically alone, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module.

[0207] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a terminal device (which may be a personal computer, a network device, etc.) or a processor to execute all or part of the steps of the method in each embodiment of the present application. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0208] An embodiment of the present application also provides a calibration device. Figure 16 Exemplarily, a possible architecture diagram of the calibration device is provided.

[0209] The modeling device includes a memory 1601, a processor 1602, a communication interface 1603, and a bus 1604. Among them, the memory 1601, the processor 1602, and the communication interface 1603 are communicatively connected to each other through the bus 1604.

[0210] The memory 1601 can be a ROM, a static storage device, a dynamic storage device, or a RAM. The memory 1601 can store a program. When the program stored in the memory 1601 is executed by the processor 1602, the processor 1602 and the communication interface 1603 are used to execute the foregoing Figure 9 , Figure 11 , Figure 13 and Figure 14 the methods shown, or implement the functions of the foregoing Figure 15 the device shown. The memory 1601 can also store point cloud data and image frame data.

[0211] The processor 1602 can be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0212] The processor 1602 can also be an integrated circuit chip with signal processing capabilities. During implementation, some or all of the functions of the modeling device of this application can be completed by the integrated logic circuit in the hardware of the processor 1602 or by instructions in software form. The foregoing processor 1602 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit, a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the foregoing embodiments of this application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of this application can be directly implemented by a hardware decoding processor or executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc.

[0213] The communication interface 1603 uses a transceiver module such as, but not limited to, a transceiver to implement communication between the calibration device and other devices or communication networks. For example, point cloud data, etc. can be obtained through the communication interface 1603.

[0214] The bus 1604 may include paths for transmitting information between various components of the modeling device (e.g., the memory 1601, the processor 1602, the communication interface 1603).

[0215] The descriptions of the processes corresponding to the above respective drawings have different focuses. For parts not detailed in a certain process, reference may be made to the relevant descriptions of other processes.

[0216] This application provides a computer-readable storage medium, including computer instructions, which when run by a processor, cause the processor to execute the calibration method described in the embodiments of this application.

[0217] This application provides a computer program product, the computer program product includes a computer program, which when run on a processor, causes the processor to execute the calibration method described in the embodiments of this application.

[0218] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a server or a terminal, the processes or functions described in the embodiments of this application are generated in whole or in part. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial optical cable, optical fiber, digital subscriber line) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by the server or the terminal, or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, and a magnetic tape, etc.), an optical medium (such as a digital video disk (DVD), etc.), or a semiconductor medium (such as a solid-state drive, etc.).

[0219] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device generate means for implementing the specified functions in one or more of the processes Figure 1 one or more of the processes and / or blocks Figure 1 or means for implementing the specified functions in one or more of the blocks.

[0220] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. A calibration method, characterized in that, Including: Obtaining point cloud data of at least one calibration target collected by a lidar on the vehicle and image frame data of the at least one calibration target collected by a camera on the vehicle; each calibration target of the at least one calibration target includes at least one first type calibration object and at least one second type calibration object; Determining the coordinates of the at least one first type calibration object in the lidar coordinate system according to the point cloud data, and determining the coordinates of the at least one second type calibration object in the image coordinate system of the camera according to the image frame data; Calibrating the first external parameter of the lidar relative to the vehicle according to the position coordinates of the at least one first type calibration object in the world coordinate system and the position coordinates of the at least one first type calibration object in the lidar coordinate system, and calibrating the second external parameter of the camera relative to the vehicle according to the position coordinates of the at least one second type calibration object in the world coordinate system and the position coordinates of the at least one second type calibration object in the image coordinate system of the camera; Wherein, the world coordinate system is established with the vehicle as a reference.

2. The method according to claim 1, characterized in that, The calibration area of the first type calibration object is a set geometric shape.

3. The method according to claim 2, wherein The calibration area is a reflector, or the calibration area is a calibration hole.

4. The method according to claim 3, wherein The reflector is circular, square or triangular.

5. The method according to claim 3, wherein The calibration hole is circular, square or triangular.

6. The method according to claim 3 or 5, characterized in that, The inner wall of the calibration hole is in a chamfered shape.

7. The method according to any one of claims 2-6, characterized in that, The position coordinates of the first type calibration object in the lidar coordinate system are the coordinates of the geometric center of the set geometric shape in the lidar coordinate system.

8. The method according to claim 7, characterized in that, Determining the position coordinates of the at least one first type calibration object in the lidar coordinate system according to the point cloud data includes: Determining the point cloud data of the region of interest from the point cloud data, where the region of interest includes the first type calibration object; Determining the edge points with significant features in the point cloud data of the region of interest; Mapping the edge points with significant features to a two-dimensional image coordinate system to obtain a two-dimensional image; Performing template segmentation on the two-dimensional image to obtain at least one light cluster region, where the at least one light cluster region corresponds one-to-one to the at least one first type calibration object; the light cluster region is used to indicate the interval range of the set geometric shape on the corresponding first type calibration object in the two-dimensional image; Performing non-maximum suppression on each of the at least one light cluster regions respectively, and processing the at least one light cluster region after non-maximum suppression to obtain the edge points of the set geometric shape on the at least one first type calibration object; Respectively determining the coordinates of the geometric midpoint of the set geometric shape on the at least one first type calibration object in the lidar coordinate system according to the coordinates of the edge points of the set geometric shape on the at least one first type calibration object in the lidar coordinate system.

9. The method according to claim 7 or 8, characterized in that, The number of the first type calibration objects is N, where N is an integer greater than 1. Calibrating the first external parameter of the lidar relative to the vehicle according to the position coordinates of the at least one first type calibration object in the world coordinate system and the position coordinates of the at least one first type calibration object in the lidar coordinate system includes: Based on the coordinates of the geometric centers of the N first-type calibration objects in the world coordinate system respectively, and the coordinates of the geometric centers of the N first-type calibration objects in the lidar coordinate system respectively, solve the first mapping equation to obtain the first extrinsic parameters of the lidar relative to the vehicle; Among them, the first mapping equation is established based on the mapping relationship between the world coordinate system and the lidar coordinate system.

10. The method according to claim 9, wherein The method further includes: After determining the first extrinsic parameters, based on the coordinates of the geometric centers of the N first-type calibration objects in the world coordinate system respectively, and the coordinates of the geometric centers of the N first-type calibration objects in the lidar coordinate system respectively, and using the first global cost function to optimize the first extrinsic parameters to obtain the optimized first extrinsic parameters; The first global cost function is used to characterize the error between the coordinates of the geometric center of the first-type calibration object in the world coordinate system and the coordinates of the geometric center of the first-type calibration object in the lidar coordinate system transformed to the world coordinate system based on the first extrinsic parameters.

11. The method according to any one of claims 1-10, characterized in that, The calibration area of the second-type calibration object is a set pattern, and the set pattern is composed of points and lines.

12. The method according to claim 11, wherein Determining the position coordinates of the at least one second-type calibration object in the image coordinate system of the camera according to the image frame data includes: Using an edge detection algorithm to detect the image data of the set pattern from the image frame data; Detecting at least one calibration point and / or at least one calibration line in the set pattern according to the image data of the second-type calibration object to obtain the coordinates of the at least one calibration point and / or at least one calibration line in the image coordinate system of the camera.

13. The method according to claim 12, wherein The calibration point is a corner point of the set pattern, and the calibration line is at least one edge line of the set pattern. Detecting the calibration point and / or calibration line in the set pattern according to the image data of the second-type calibration object includes: Obtaining multiple corner points and / or multiple edge lines in the set pattern according to the image data of the set pattern; Determining the descriptors of the multiple corner points and / or the descriptors of the multiple edge lines; Screening out at least one corner point as the calibration point from the multiple corner points according to the descriptors of the multiple edge points, and / or, screening out at least one edge line as the calibration line from the multiple edge lines according to the descriptors of the multiple edge lines, so as to obtain the coordinates of the at least one corner point as the calibration point and / or at least one edge line as the calibration line in the image coordinate system of the camera.

14. The method according to claim 13, wherein The number of the second-type calibration objects is M, and M is an integer greater than 1. Calibrating the second extrinsic parameters of the camera relative to the vehicle according to the position coordinates of the at least one second-type calibration object in the world coordinate system and the coordinates of the at least one second-type calibration object in the image coordinate system of the camera includes: Based on the coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second-type calibration objects in the world coordinate system, and the coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second-type calibration objects in the image coordinate system of the camera, solve the second mapping equation to obtain the second extrinsic parameters of the camera relative to the vehicle; Wherein, the second mapping equation is established based on the mapping relationship between the world coordinate system and the image coordinate system of the camera.

15. The method according to claim 14, wherein The method further includes: After determining the second extrinsic parameters, based on the coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second-type calibration objects in the world coordinate system, and the coordinates of at least one calibration point and / or at least one calibration line included in the set pattern on the M second-type calibration objects in the lidar coordinate system, and using the second global cost function to optimize the second extrinsic parameters to obtain the optimized second extrinsic parameters; The second global cost function is used to characterize the error between the coordinates of the calibration mark in the image coordinate system of the camera and the coordinates obtained by converting the coordinates of the calibration mark in the world coordinate system to the image coordinate system using the second extrinsic parameters, and the calibration mark includes at least one calibration point and / or at least one calibration line.

16. The method according to any one of claims 1 to 15, characterized in that, Obtaining the point cloud data of at least one calibration target collected by the lidar on the vehicle and the image frame data of the at least one calibration target collected by the camera on the vehicle includes: Obtaining a solution set and a verification set, where the solution set includes the point cloud data corresponding to L1 timestamps and the image frame data corresponding to the L1 timestamps; the verification set includes the point cloud data corresponding to L2 timestamps and the image frame data corresponding to the L2 timestamps; After obtaining the first extrinsic parameters and the second extrinsic parameters using the point cloud data corresponding to the i-th timestamp among the L1 timestamps and the image data corresponding to the i-th timestamp, use the verification set to verify the first extrinsic parameters and the second extrinsic parameters; When the verification does not meet the calibration conditions, continue to use the point cloud data corresponding to the (i + 1)-th timestamp among the L1 timestamps and the image frame data corresponding to the (i + 1)-th timestamp to adjust the first extrinsic parameters and the second extrinsic parameters until the verification meets the calibration conditions; When the verification meets the calibration conditions, use the first extrinsic parameters that meet the calibration conditions as the extrinsic parameters of the lidar relative to the vehicle obtained by calibration, and use the second extrinsic parameters that meet the calibration conditions as the extrinsic parameters of the camera relative to the vehicle obtained by calibration.

17. A calibration device, characterized in that, Including a memory and one or more processors; wherein, the memory is used to store computer program code, and the computer program code includes computer instructions; when the computer instructions are executed by the processor, the device is caused to execute the method according to any one of claims 1-16.

18. A computer-readable storage medium, characterized in that, Comprising computer instructions, when the computer instructions are run on the calibration device, enabling the calibration device to execute the method according to any one of claims 1-16.

19. A vehicle, characterized in that, The vehicle comprises an external parameter calibration device for an in-vehicle radar according to claim 17.

20. A calibration device, characterized in that, The calibration device comprises at least one reflector and at least one calibration pattern; The at least one reflector is used for calibrating a lidar on the vehicle; The at least one calibration pattern is used for calibrating a camera on the vehicle.

21. The calibration device according to claim 20, wherein, The calibration pattern is a checkerboard or a two-dimensional code.

Images