Calibration method and device and storage medium

Abstract

The invention provides a calibration method and device and a storage medium. According to the main technical scheme, the method comprises the following steps: respectively acquiring images and point cloud data about a calibration plate acquired by a camera and a laser radar, wherein the calibration plate comprises a first feature point and a second feature point; determining a pixel coordinate system coordinate of the first feature point according to the image; determining a laser radar coordinate system coordinate of the second feature point according to the point cloud data; determining a first transformation relation between the calibration plate coordinate system and the laser radar coordinate system according to the calibration plate coordinate system coordinate and the laser radar coordinate system coordinate of the second feature point; determining a laser radar coordinate system coordinate of the first feature point according to the calibration plate coordinate system coordinate of the first feature point and the first transformation relation; and performing parameter calibration on the camera and the laser radar according to the pixel coordinate system coordinate and the laser radar coordinate system coordinate of the first feature point. According to the invention, the accuracy, robustness and universality of calibration can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a calibration method, device, and storage medium. Background Art

[0002] Sensor calibration in the autonomous driving field involves calibrating the internal and external parameters of sensors such as cameras, Lidar (Light Detection and Ranging), millimeter-wave radar, inertial navigation, and integrated navigation. This ensures the coordinates of each sensor are unified and provides reliable input for perception algorithms. Calibration methods for these sensors include target-based calibration. In these target-based calibration methods, effective feature extraction and correlation of the same features of the sensor to be calibrated (such as Lidar) are the basis for optimizing the internal and external parameters.

[0003] At present, target-based calibration methods have few associated features added to the solution of the spatial transformation matrix. For example, feature points such as marked corner points cannot be added to the solution of the spatial transformation matrix. As a result, calibration schemes such as the calibration of camera extrinsic parameters and the joint calibration of cameras and other sensors have low robustness and versatility. At the same time, the success rate and accuracy of calibration are difficult to guarantee, making them unsuitable for mass production calibration in the field of autonomous driving. Summary of the Invention

[0004] In view of this, the present disclosure provides a calibration method, apparatus, and storage medium to improve the robustness, accuracy, and versatility of calibration.

[0005] According to a first aspect of the present disclosure, a calibration method is provided, the method comprising: respectively acquiring images and point cloud data of a calibration plate collected by a camera and a lidar, the calibration plate being provided with a mark, the calibration plate comprising a first feature point about the mark and a second feature point about a structural contour of the calibration plate; determining the pixel coordinate system coordinates of the first feature point according to the image; determining the lidar coordinate system coordinates of the second feature point according to the point cloud data; determining a first transformation relationship between the calibration plate coordinate system and the lidar coordinate system according to the calibration plate coordinate system coordinates and the lidar coordinate system coordinates of the second feature point; determining the lidar coordinate system coordinates of the first feature point according to the calibration plate coordinate system coordinates and the first transformation relationship; and calibrating parameters of the camera and the lidar according to the pixel coordinate system coordinates and the lidar coordinate system coordinates of the first feature point.

[0006] According to a second aspect of the present disclosure, a calibration device is provided, comprising: one or more processors, and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, cause the processor to execute the method described in the first aspect above.

[0007] According to a third aspect of the present disclosure, a computer-readable storage medium storing a program is provided, wherein the program includes instructions, which, when executed by one or more processors of a computing device, cause the computing device to execute the method described in the first aspect above.

[0008] It can be seen from the above technical solution that the present disclosure obtains the images and point cloud data collected by the camera and the lidar, obtains the spatial transformation relationship (i.e., the first transformation relationship) between the calibration plate coordinate system and the lidar coordinate system based on the lidar coordinates of the second feature point about the calibration plate structure outline extracted from the point cloud data and the calibration plate coordinate system coordinates of the second feature point, and then determines the lidar coordinates of the first feature point marked on the calibration plate through the calibration plate coordinates and the spatial transformation relationship. Combined with the pixel coordinates of the first feature point extracted from the image, the feature points of each mark on the calibration plate can be applied to the calibration of the camera and the lidar, thereby realizing the rapid calibration of the internal and external parameters of the camera and the lidar, without manual participation and without the need to add prior data such as maps, and the versatility, accuracy and robustness of the calibration are effectively improved. In addition, actual tests have found that compared with other existing calibration methods, the embodiments of the present disclosure have more advantages in versatility, accuracy, robustness and execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 A system architecture diagram applicable to the embodiments of the present disclosure;

[0011] Figure 2 A schematic diagram of a flow chart of a calibration method provided in an embodiment of the present disclosure;

[0012] Figure 3 This is an example diagram of a calibration plate involved in an embodiment of the present disclosure;

[0013] Figure 4 An example diagram of the calibration plate coordinate system and feature points involved in the embodiments of the present disclosure;

[0014] Figure 5 This is an example diagram of the laser beam line of the laser radar involved in the embodiments of the present disclosure;

[0015] Figure 6 A schematic diagram of the laser beam angle at a point in a point cloud according to an embodiment of the present disclosure;

[0016] Figure 7 An example diagram is selected for the first reference point involved in the embodiment of the present disclosure;

[0017] Figure 8 A schematic diagram of a process for calibrating a camera and a lidar according to an embodiment of the present disclosure;

[0018] Figure 9 Schematic diagram of an exemplary implementation process for calibrating a camera and a lidar based on the initial values ​​of the camera's intrinsic parameters and extrinsic parameters in an embodiment of the present disclosure;

[0019] Figure 10 Another schematic diagram of a process for calibrating a camera and a lidar according to an embodiment of the present disclosure;

[0020] Figure 11 Another schematic diagram of a flow chart of a calibration method provided in an embodiment of the present disclosure;

[0021] Figure 12 A modular structural block diagram of the calibration device provided in an embodiment of the present disclosure;

[0022] Figure 13 A schematic block diagram of a calibration device in the form of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0024] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a," "an," "the," and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0026] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0027] Explanation of terms:

[0028] Calibration room: The calibration room is more conducive to forming a standardized calibration operation process. By arranging calibration plates (also called targets) suitable for feature extraction of various types of sensors in the calibration room, the robustness and accuracy of sensor parameter calibration can be improved. It can also be connected to the vehicle production process to realize sensor calibration before the vehicle leaves the factory, ensuring the safety of autonomous driving.

[0029] Iterative Closest Point (ICP): A point cloud matching algorithm. Given the coordinates of a 3D point in two three-dimensional space coordinate systems and their corresponding relationship, the transformation relationship between the two three-dimensional space coordinate systems can be solved through ICP. The transformation relationship includes a transformation matrix, and the transformation matrix includes a rotation matrix and a translation matrix. In the disclosed embodiment, the point correspondence relationship can be directly obtained by sequentially reading the coordinates of the calibration plate coordinate system and the laser radar coordinate system coordinates of the first feature point according to the plate identification and the marking identification. There is no need to perform processes such as finding the closest point or the nearest neighbor point. This can greatly simplify the construction process of the ICP problem, reduce computational complexity and amount of calculations, reduce the use of computing resources, reduce hardware costs, and improve processing efficiency.

[0030] PnP (Perspective-n-Point): is a method for solving the motion of three-dimensional to two-dimensional point pairs. When the three-dimensional coordinates of n (n is an integer greater than 1) feature points in the world coordinate system and the pixel coordinates of these points are known, the camera's pose in the world coordinate system can be estimated through PnP. If the world coordinate system is selected as the lidar coordinate system, PnP can be used to solve the camera's pose in the lidar coordinate system, that is, the pose of the camera coordinate system relative to the lidar coordinate system. This pose is the transformation relationship between the camera coordinate system and the lidar coordinate system. In the embodiment of the present disclosure, the pixel coordinate system coordinates and the lidar coordinate system coordinates of the first feature point are sequentially read according to information such as plate identification and mark identification to directly obtain the point correspondence relationship between the pixel coordinate system and the lidar coordinate system. There is no need to perform the calculation process to determine the point correspondence relationship. This can greatly simplify the construction process of the PnP problem, reduce computational complexity and amount of calculation, reduce the use of computing resources, reduce hardware costs, and improve processing efficiency.

[0031] Camera intrinsic parameters: These include the modeling parameters for the projection process from the camera coordinate system to the pixel coordinate system. These parameters vary depending on the camera type and projection model. For example, for the pinhole model used with long / short focal length cameras, the camera intrinsic parameters may include fx, fy, cx, cy, k1, k2, p1, p2, and k3. fx and fy represent the scale factors of the camera in the horizontal and vertical axes of the pixel coordinate system, respectively. cx and cy represent the coordinates of the camera's principal point in the pixel coordinate system. k1, k2, and k3 represent radial distortion parameters, and p1 and p2 represent tangential distortion parameters. For another example, the Scaramuzza model used with fisheye cameras and the Double Sphere model used with fisheye cameras each have different camera intrinsic parameters.

[0032] Camera extrinsics: This refers to the transformation relationship between the camera coordinate system and other coordinate systems. This transformation relationship includes the transformation matrix, which includes the translation matrix and the rotation matrix. Among these other coordinate systems, these other coordinate systems include but are not limited to the world coordinate system, the vehicle coordinate system, the calibration coordinate system, and other sensor coordinate systems. Other sensor coordinate systems include but are not limited to the LiDAR coordinate system, the millimeter-wave radar coordinate system, and other camera coordinate systems.

[0033] Pixel coordinate system: A two-dimensional rectangular coordinate system perpendicular to the camera optical axis. The two-dimensional rectangular coordinate system includes the horizontal axis u and the vertical axis v. The plane formed by the u and v axes can be parallel to the plane formed by the x and y axes in the camera coordinate system.

[0034] Camera coordinate system: A three-dimensional rectangular coordinate system centered on the camera. The positive z-axis can be set to point outside the lens along the camera optical axis, and the x-axis and y-axis are parallel to the u-axis and v-axis of the pixel coordinate system.

[0035] LiDAR coordinate system: a three-dimensional rectangular coordinate system centered on the LiDAR.

[0036] Calibration plate coordinate system: Each calibration plate has its own calibration plate coordinate system. The calibration plate coordinate system of each calibration plate is a three-dimensional rectangular coordinate system with the geometric center of the calibration plate or other selected points on the calibration plate as the origin. The plane formed by the horizontal and vertical axes of the three-dimensional rectangular coordinate system is the calibration plate plane, and the z-axis of the three-dimensional rectangular coordinate system is perpendicular to the calibration plate plane.

[0037] The common viewing area of ​​the camera and the lidar: the overlapping area of ​​the camera's field of view and the lidar's field of view.

[0038] High reflectivity points: Feature points with high laser intensity values ​​in the point cloud data. High reflectivity points on the calibration plate can be feature points with higher reflectivity than surrounding points on the calibration plate. High reflectivity points can be formed by placing high-reflectivity materials at specific locations on the calibration plate.

[0039] Exemplary system architecture

[0040] In order to facilitate the understanding of the embodiments of the present disclosure, first Figure 1 The system architecture applicable to the embodiments of the present disclosure is described. Figure 1 An exemplary system architecture diagram to which the present disclosure can be applied is shown in FIG. Figure 1 The system mainly includes sensors such as cameras, Lidar (Light Detection and Ranging), millimeter-wave radar, inertial navigation, integrated navigation, and calibration devices. Each sensor communicates with the calibration device. The camera is used to capture images of the environment within its field of view, and the calibration device can be used to extract feature points based on the images provided by the camera and calibrate the camera based on the feature points.

[0041] The camera may be, but is not limited to, a short / telephoto camera of a pinhole model, a fisheye camera of a Scaramuzza model, a fisheye camera using a Double Sphere model, or the like.

[0042] The above-mentioned system is mounted on different platforms according to the needs of different scenarios to realize the calibration of sensors. For example, the above-mentioned system can be set on a mobile platform such as a vehicle or a robot, and the mobile platform is stopped at a specified position in a calibration scene such as a calibration room (for example, the center position of the calibration room, etc.) and aligned so that the calibration plates placed in the calibration room are in the common viewing area of ​​the sensor, so as to calibrate the sensor. For another example, the above-mentioned system can be set on a mobile platform such as a vehicle or a robot, and the mobile platform is stopped at any position in any scene, and the calibration plate is arranged in the common viewing area of ​​the sensor, and the calibration of the sensor is realized by moving the calibration plate or switching the posture in the common viewing area.

[0043] As a possible implementation, the calibration device can be installed on a computer terminal with strong computing power deployed on the sensor platform, or on a server deployed independently of the sensor platform. The calibration device can communicate with the sensors on the platform through various wireless communication methods, or can be interconnected with each sensor through a physical interface. The calibration device calibrates the intrinsic and / or extrinsic parameters of each sensor by acquiring, analyzing, and processing the sensor data collected by each sensor.

[0044] It should be understood that Figure 1 The number of calibration devices and sensors in the figure is only for illustration. Any number and type of sensors and calibration devices may be used as required.

[0045] Example application scenarios

[0046] The embodiments of the present disclosure are applicable to the calibration of various types of sensors. For example, the present disclosure can realize the calibration of various types of sensors even without prior information such as high-definition maps. For another example, the present disclosure can also be applied to the unified calibration of multiple sensors such as cameras, lidars, millimeter-wave radars, etc. For another example, the embodiments of the present disclosure are applicable to application scenarios such as autonomous driving with a large number of sensor types and high accuracy requirements. In particular, since the embodiments of the present disclosure do not require prior information such as maps and have low requirements for the calibration environment and are not restricted to calibration places such as calibration rooms, they are particularly suitable for after-sales calibration scenarios of sensors for autonomous driving vehicles.

[0047] Because data collected by a single sensor cannot accurately describe environmental information, multi-sensor fusion technology has emerged. Multi-sensor fusion uses computer technology to automatically analyze and synthesize information and data from multiple sensors or sources according to specific criteria to make the necessary decisions and estimates. Multi-sensor fusion enables data from different sensors to complement each other and is widely used in remote sensing, robotics, autonomous driving, 3D space reconstruction, and other fields.

[0048] In multi-sensor fusion technology, the fusion of lidar and cameras is widely used. Through the fusion and complementarity of lidar and cameras, it can accurately perceive the color three-dimensional data of space, and is widely used in industrial control, autonomous driving, robotics and other fields.

[0049] In the data fusion of cameras and lidars, accurate and stable external calibration between sensors is crucial. That is, it is necessary to calibrate the parameters of the camera and lidar through the technical solution of the embodiment of the present disclosure to find a suitable transformation relationship, so as to realize the mutual conversion between the data points of the two coordinate systems of the lidar and the camera, and then obtain accurate and complete information about the external environment or the observed target through the fusion algorithm for subsequent processing such as positioning, tracking, navigation, path planning, and behavioral decision-making.

[0050] In the disclosed embodiment, parameter calibration is to obtain the internal and external parameters of the sensor, thereby achieving coordinate unification of each sensor and providing reliable input for perception algorithms, etc.

[0051] Exemplary Methods

[0052] Figure 2 This is a flow chart of the calibration method provided in the embodiment of the present disclosure. The method can be performed by Figure 1 The calibration device in the system shown is performed. Figure 2 , the method may include the following steps:

[0053] Step 201: acquiring images and point cloud data of a calibration plate collected by a camera and a lidar, respectively, wherein the calibration plate is provided with a mark, and the calibration plate includes first feature points of the mark and second feature points of a structural profile of the calibration plate;

[0054] Step 203: Determine the pixel coordinate system coordinates of the first feature point according to the image;

[0055] Step 205: Determine the laser radar coordinate system coordinates of the second feature point based on the point cloud data;

[0056] Step 207: determining a first transformation relationship between the calibration plate coordinate system and the laser radar coordinate system according to the calibration plate coordinate system coordinates and the laser radar coordinate system coordinates of the second feature point;

[0057] Step 209: Determine the laser radar coordinate system coordinates of the first feature point based on the calibration plate coordinate system coordinates and the first transformation relationship;

[0058] Step 211: Calibrate the camera and the lidar according to the pixel coordinate system coordinates and the lidar coordinate system coordinates of the first feature point.

[0059] The present invention obtains images and point cloud data collected by cameras and laser radars, obtains the spatial transformation relationship (i.e., the first transformation relationship) between the calibration plate coordinate system and the laser radar coordinate system based on the laser radar coordinates of the second feature point of the calibration plate structure outline extracted from the point cloud data and the calibration plate coordinate system coordinates of the second feature point, and then determines the laser radar coordinate system coordinates of the first feature point of each mark on the calibration plate through the calibration plate coordinates and the spatial transformation relationship. Combined with the pixel coordinate system coordinates of the first feature point extracted from the image, the feature points of each mark on the calibration plate can be applied to the calibration of the camera and laser radar, thereby realizing rapid calibration of the internal and external parameters of the camera and laser radar, without manual participation and without the need to add prior data such as maps, and the versatility, accuracy and robustness of the calibration are all improved. In addition, actual tests have found that compared with other existing calibration methods, the embodiments of the present invention have more advantages in terms of versatility, accuracy, robustness and execution efficiency. The calibration plate, mark, calibration hole, various feature points and coordinate system involved in the embodiments of the present invention are described in detail below.

[0060] The calibration plate, also known as a target, is primarily used for sensor calibration. In some embodiments of the present disclosure, one or more marks may be provided on the calibration plate, each with a corresponding marking identifier. In some other embodiments of the present disclosure, the calibration plate may further include one or more calibration holes.

[0061] In some embodiments, to better capture effective features during calibration, the marks on the calibration plate may be arranged symmetrically, and the calibration holes on the calibration plate may also be arranged symmetrically.

[0062] The mark can be printed, drawn, or pasted on the calibration plate. The mark can be a regular shape such as a quadrilateral, triangle, circle, or other. The mark can also be an irregular shape. In some embodiments of the present disclosure, the mark can be, but is not limited to, a QR code, a picture, etc., and each mark has a unique mark identifier. For example, the mark can be, but is not limited to, an ArUco marker (arucomarker), i.e., an ArUco QR code.

[0063] The calibration hole is preferably a regular shape. For example, the calibration hole can be, but is not limited to, a circular through hole (referred to as a circular hole), a square through hole, a rectangular through hole, or a through hole of any other shape.

[0064] Figure 3 An example diagram of a calibration plate according to an embodiment of the present disclosure is shown. Figure 3 The calibration plate 300 is provided with four symmetrically distributed marks and four symmetrically distributed circular holes. The four marks can be QR codes or pictures. Each mark can have the same size, and each circular hole can have the same size.

[0065] Different types of calibration plates can be designed as needed to calibrate different sensors in different scenarios. To facilitate feature capture and camera calibration calculations, different calibration plates, different markings on the plates, and different calibration holes on the plates can be of the same or different sizes, and this disclosure does not impose any restrictions on this.

[0066] The first feature point of the mark can be pre-selected as needed. In order to more accurately capture the features of the mark, the first feature point of the mark can be a feature point on the outline of the mark, such as a corner point of the mark (ie, a vertex of the mark). See below Figure 4 For example, the four corner points p5 to p8 of the mark can be selected as the first feature points of the mark.

[0067] The second characteristic points of the structural contour of the calibration plate can also be selected according to actual needs. In one implementation, the second characteristic points of the structural contour of the calibration plate can include at least one or more of the following: the first reference point of the outer contour of the calibration plate, the second reference point of the calibration hole in the calibration plate, and the high reflection point of the calibration plate. To facilitate the accurate capture of characteristic points by the laser radar, the first reference point of the outer contour of the calibration plate can include but is not limited to the four edge corner points of the calibration plate. To facilitate the accurate capture of characteristic points by the laser radar, the second reference point of the calibration hole in the calibration plate can include but is not limited to the corner points and center points of the calibration hole. See Figure 4For example, the calibration plate is rectangular, the calibration hole is a circular hole, the first reference point of the outer contour of the calibration plate may include the four edge corner points p1 to p4 of the calibration plate, and the second reference point of the calibration hole may include the center point p9 of the calibration hole.

[0068] The attribute information of the calibration plate may include, but is not limited to, the plate identification, dimensions, number of marks on the calibration plate, mark identification of each mark, mark size and mark position, number of holes on the calibration plate, calibration hole identification of each calibration hole, calibration hole size and calibration hole position, etc. The dimensions of the calibration plate may include the length and width of the calibration plate. The position of the calibration hole may include information such as the distance from the calibration hole to the adjacent edge of the calibration plate, the positional relationship of the calibration hole relative to a specific corner point of the calibration plate, the distance between the calibration hole and the adjacent calibration hole, and the distance between the calibration hole and the adjacent mark. The position of the mark may include information such as the distance from the mark to the adjacent edge of the calibration plate, the distance between the mark and the adjacent calibration hole, and the distance between the mark and the adjacent mark.

[0069] Taking a quadrilateral mark as an example, the size of the mark may include the length and width of the quadrilateral mark. The size of the calibration hole may include the diameter or radius of the circular calibration hole, or the length and width of the quadrilateral hole.

[0070] The shape, size, and distribution of the marks and calibration holes of the calibration plate can be determined using the above attribute information.

[0071] In one implementation, various attribute information of the calibration plate can be stored in association with plate identification, mark identification, etc., so that the attribute information of the required calibration plate, mark and calibration hole can be queried efficiently, quickly and accurately.

[0072] In some embodiments of the present disclosure, the above-mentioned calibration method may further include: establishing a calibration plate coordinate system and a structure object of the calibration plate, and storing the calibration plate coordinate system coordinates of each first feature point and each second feature point on the calibration plate in the structure object according to the attribute information of the calibration plate, so that the calibration plate coordinate system coordinates of the required feature points can be directly read during calibration, thereby speeding up the calibration process, improving calibration efficiency, reducing repeated operations, and reducing computational complexity.

[0073] Specifically, for each calibration plate, a calibration plate coordinate system is first constructed for that plate. Based on the plate's attribute information, the calibration plate coordinates of all first and second feature points on the plate are calculated and written into the plate's structure object. For example, the calibration plate coordinates of the first feature point are the three-dimensional coordinates of the calibration plate to which the first feature point belongs in the calibration plate coordinate system.

[0074] The calibration plate coordinate system can be flexibly constructed according to actual needs. For example, the calibration plate coordinate system can be constructed with the center of the calibration plate as the origin, the xy axis plane parallel to the calibration plate plane, and the z axis perpendicular to the calibration plate screen. Figure 4 An example diagram of the calibration plate coordinate system of an embodiment of the present disclosure is shown. Figure 4 The origin of the calibration plate coordinate system xyz is selected as the geometric center O of the calibration plate. The calibration plate contains a quadrilateral mark and a circular calibration hole. The second characteristic points of the calibration plate can include the four edge corner points p1 to p4 of the calibration plate and the center point p9 of the calibration hole. The first characteristic points of the mark can include the four corner points p5 to p8 of the mark. It should be noted that Figure 4 This is just for illustration, and the construction method of the calibration plate coordinate system in the embodiment of the present disclosure is not limited to this.

[0075] In one implementation, the calibration plate coordinate system coordinates of the first feature point and the second feature point on the calibration plate can be stored in a pre-agreed fixed order so as to directly read the calibration plate coordinate system coordinates of one or more required feature points.

[0076] In one example, the calibration plate coordinates of each first feature point on the mark can be stored in association with the corresponding mark identifier and plate identifier in order according to the positional order of the marks on the calibration plate. For each mark, the calibration plate coordinates of each first feature point on the mark can be stored in association with the corresponding mark identifier and plate identifier in order according to the positional order of the first feature point on the mark.

[0077] In one example, the calibration plate coordinate system coordinates of each second feature point in the calibration plate can be associated and stored with the corresponding plate identifier in sequence according to the position sequence of each calibration hole in the calibration plate. Specifically, the calibration plate coordinate system coordinates of each second reference point in the calibration plate can be associated and stored with the corresponding plate identifier in sequence, and the calibration plate coordinate system coordinates of each first reference point in the calibration plate can be associated and stored with the corresponding plate identifier in sequence according to the position sequence of each first reference point on the outer contour of the calibration plate.

[0078] by Figure 3 For example, taking the four corner points of the mark as the first feature points, and the four corner points of the calibration plate 300 and the centers of the circular calibration holes on the calibration plate 300 as the second feature points, the calibration plate coordinate system coordinates of the first feature points and the second feature points can be associated and stored with the plate identification, mark identification, etc. in a clockwise order.

[0079] For each mark on the calibration plate, the calibration plate coordinates of the corner points of each mark can be stored in the clockwise order of "left, top, right, bottom". At the same time, for each mark, the calibration plate coordinates of the four corner points on the mark are stored in the clockwise order of "top left, top right, bottom right, bottom left" in association with the corresponding plate identifier, mark sequence number, and mark identifier.

[0080] For the four corner points on the calibration plate, the calibration plate coordinate system coordinates of the four corner points on the calibration plate, the sequence numbers of the four corner points, and the corresponding plate identifiers can also be associated and stored in the clockwise order of "upper left corner, upper right corner, lower right corner, lower left corner".

[0081] For each calibration hole on the calibration plate, the calibration plate coordinate system coordinates of the center point of each calibration hole, the sequence number of each center point and the corresponding plate identification can also be associated and stored in the clockwise order of "left, top, right, bottom".

[0082] In one example, the calibration board structure object may be, but is not limited to, a Board structure object. Using the Board structure object, the calibration board coordinate system coordinates of each feature point can be conveniently associated and stored with a board identifier, a marker identifier, and the like. It should be understood that the calibration board structure object may also employ any other type of structure applicable to the present disclosure, and the present disclosure does not impose any limitation thereto.

[0083] By adopting the above storage method, the calibration plate coordinate system coordinates of a specific feature point on a calibration plate can be conveniently read efficiently and accurately through the plate identification and marking identification of the calibration plate.

[0084] The calibration method of the embodiment of the present disclosure may be implemented using a single calibration board, or may be performed in a calibration location such as a calibration room containing multiple calibration boards.

[0085] If the calibration site is limited and calibration cannot be performed in a specific calibration site such as a calibration room, the calibration plate can be moved during the acquisition process of step 201 so that the acquired images and point cloud data meet the spatial distribution requirements of the feature points required for calibration. Specifically, the acquisition of step 201 is performed simultaneously during the process of moving the calibration plate. Assuming that the calibration plate has n positions (n ​​is an integer greater than 1) by moving the calibration plate, different positions of the n positions have different heights, angles and / or distances relative to the loading platforms of the laser radar and the camera and are all located in the common view area of ​​the camera and the laser radar, at this time, the images and point cloud data about the calibration plate acquired in step 201 may include: n image sequences and n point cloud sequences about the calibration plate, and the n image sequences correspond one-to-one to the n point cloud sequences and the n positions of the calibration plate. Therefore, calibration can be completed by moving a calibration plate multiple times. There is no need to rely on complex pre-calibration preparations such as high-precision maps for joint calibration or calibration rooms for production line calibration. It has the advantages of flexibility, efficiency and easy deployment, and is suitable for after-sales calibration scenarios or other temporary collision test scenarios.

[0086] Each image sequence may include m consecutive image frames, where m is an integer greater than 1. The point cloud sequence may include m consecutive frames of point cloud data. Alternatively, each image sequence may include one or more image frames extracted from the m consecutive image frames, and the point cloud sequence may include one or more point cloud frames extracted from the m consecutive frames of point cloud data. In particular, the image frames in the image sequence corresponding to the same position in the n positions and the point cloud frames in the point cloud sequence are at least partially acquired at the same time, that is, the image sequence and the point cloud sequence corresponding to the same position in the n positions are acquired synchronously.

[0087] It should be noted that, in the embodiments of the present disclosure, "consistent acquisition time" means that the acquisition time difference is within a predetermined range. If the acquisition time difference between the two is within this predetermined range, the two can be considered synchronized. The predetermined range can be flexibly set according to actual needs. For example, the predetermined range can be 1 second or other values.

[0088] For example, during the acquisition process in step 201, the calibration plate can be held and slowly moved within the common view area to obtain image sequences and point cloud sequences at multiple locations of the calibration plate. To ensure the accuracy of the calibration results, the calibration plate can be moved or rotated within the common view area at different positions, heights, distances, and angles, so that the image sequences and point cloud sequences collected at multiple locations can better meet the calibration requirements for the spatial distribution of feature points.

[0089] To ensure that at least some of the frames in the image sequence and point cloud sequence are time-synchronized, the LiDAR and camera to be calibrated can be time-synchronized before step 201 to obtain point cloud data that is synchronized with the timestamp of the image data. Time synchronization can be performed by, but is not limited to, soft synchronization or hard synchronization.

[0090] If calibration can be performed in a calibration room, multiple calibration plates can be placed in the room at different locations. The image and point cloud data in the calibration room can be collected in step 201. Multiple types of calibration plates can be placed in the calibration room, each at a different location and within the common view area of ​​the camera and lidar.

[0091] When calibration is performed in the calibration room, the image and point cloud data about the calibration plate obtained in step 201 are specifically: an image sequence and a point cloud sequence about the calibration room where the calibration plate is located, a plurality of calibration plates are provided in the calibration room and the positions of the plurality of calibration plates are different, each image frame in the image sequence contains part or all of the calibration plates in the calibration room, each point cloud frame in the point cloud sequence contains point cloud data of part or all of the calibration plates in the calibration room, and the image frames in the image sequence and the point cloud frames in the point cloud sequence are at least partially collected at the same time. Thus, calibration can be conveniently completed directly through the image sequence and point cloud sequence at one time, avoiding the need to constantly move the calibration plate during the calibration process, and achieving the efficiency required for production line calibration. Among them, the image sequence can include multiple continuous frames of images, or multiple frames of images with a fixed number of frames or a fixed time interval. The point cloud sequence can include multiple continuous frames of point cloud data, or multiple frames of point cloud data with a fixed number of frames or a fixed time interval. The present disclosure does not impose any restrictions on the number of frames, continuity, and interval length in image sequences and point cloud sequences.

[0092] In some embodiments, if step 201 acquires an image sequence and a point cloud sequence of the calibration room in which the calibration plate is located, the process may further include: performing target detection on each frame of the image sequence, and determining at least one frame containing the calibration plate and having a number of markers on the calibration plate greater than or equal to a first predetermined threshold. The first predetermined threshold can be set based on the detection requirements of a specific scene. For example, the first predetermined threshold can be set to the number of markers required for the minimum detection requirement of the current scene. Thus, if a frame does not contain the calibration plate or the number of markers detected on the calibration plate in the frame is less than the minimum detection requirement, the frame can be discarded. If a frame contains the calibration plate and the number of markers detected on the calibration plate in the frame is greater than or equal to the minimum detection requirement, the calibration plate in the frame can be included in the feature point extraction and calibration process. Thus, a robust verification mechanism for detecting a number of markers on a calibration plate greater than or equal to the minimum detection requirement can be utilized to discard poorly observed images, further reducing the amount of data, lowering computing resource consumption, and improving the versatility, accuracy, and robustness of the calibration.

[0093] After step 201, the image frames and point cloud frames in the image sequence and point cloud sequence can be filtered, and some of the image frames and point cloud frames can be selected to participate in the subsequent calibration process. This can not only reduce the amount of data, reduce the consumption of computing resources, and improve calibration efficiency, but also help improve the accuracy and robustness of the calibration results. For example, the image frames in the image sequence and the point cloud frames in the point cloud sequence can be filtered by selecting one frame at every predetermined time interval. Of course, other methods can also be used, and this disclosure is not limited thereto.

[0094] Steps 203 to 205 can be performed separately for the image frames in the image sequence and the corresponding point cloud frames in the point cloud sequence. In step 207, the LiDAR coordinate system coordinates of the second feature point in the multiple image frames and point cloud data are combined to construct an ICP problem to solve the first transformation relationship, thereby improving the versatility, robustness, and accuracy of the first transformation relationship. Specific details will be described below.

[0095] Steps 203 to 211 can be performed separately for each frame in the image sequence and each point cloud frame in the point cloud sequence that was acquired at the same time. In step 211, the pixel coordinates and the LiDAR coordinates of the first feature point of the multiple image frames and point cloud data are combined for calibration, thereby further improving the versatility, robustness, and accuracy of the calibration results. The details are described below.

[0096] In step 203, feature point detection and target detection can be performed on each frame of the image acquired in step 201 to obtain the pixel coordinate system coordinates and corresponding marker identifiers of the first feature points of all calibration plates in the image. The marker identifier can be used to query the corresponding plate identifier from, for example, the structure object of the calibration plate. The pixel coordinate system coordinates of each first feature point are associated with the corresponding plate identifier and marker identifier in the order of the first feature point's position on the marker. In this way, the pixel coordinate system coordinates of a specific first feature point can be directly read in step 211, thereby reducing the complexity of constructing the PnP problem, reducing the amount of calculation, saving computing resources, and improving calibration efficiency.

[0097] Specifically, after the corresponding plate identifier is queried through the mark identifier, the pixel coordinate system coordinates of the first feature point can be stored in the corresponding calibration plate structure object to achieve the associated storage of the pixel coordinate system coordinates of the first feature point with its mark identifier and plate identifier.

[0098] There are many ways to extract the laser radar coordinate system coordinates of the second feature point from the point cloud data in step 205. An exemplary method for extracting the second feature point is described in detail below.

[0099] In one implementation, an exemplary implementation process for extracting the laser radar coordinate system coordinates of the second feature point may include the following steps S11 to S13:

[0100] Step S11, identifying point cloud points corresponding to the same laser beam from the point cloud data;

[0101] In one implementation, the point cloud points corresponding to the same laser beam can be determined based on the laser beam identifier of each point cloud point in the point cloud data. The laser beam identifier can be, but is not limited to, a ring value of the laser beam emitted by the laser radar.

[0102] Figure 5 An example diagram of the laser beam path of a 16-line laser radar is shown. Figure 5 Taking a 16-line laser radar as an example, if the vertical measurement angle range of the laser radar is 30° (i.e., -15° to +15°), the vertical angle resolution is 2°, and each of the 16 laser beams in the vertical direction of the laser radar has a corresponding line value (ring) (1 to 16), and the adjacent lines are spaced at an angle of 2° in the vertical direction.

[0103] If the point cloud data carries a ring value, the current frame point cloud data collected by the lidar can be directly divided into point cloud groups corresponding to the laser beam lines of the lidar according to the ring value of the point. Each point cloud group contains all points corresponding to the same ring value in the current frame point cloud data, and all points corresponding to the same ring value also correspond to the point cloud points of the same laser beam.

[0104] In one implementation, the angle of the corresponding laser beam can be determined based on the coordinates of each point cloud point in the point cloud data, and points with similar angles can be identified as point cloud points of the same laser beam. Similar angles mean that the angle of the corresponding laser beam is less than or equal to a third predetermined threshold. This third predetermined threshold can be determined based on the vertical angle resolution of the lidar or set as an empirical value.

[0105] Specifically, if the point cloud data does not contain line values, the LiDAR coordinates of each point in the point cloud can be used to calculate the angle between the laser beam corresponding to that point and the horizontal plane. This angle is used as the angle of the laser beam corresponding to that point. Points with similar corresponding laser beam angles are classified as points corresponding to the same laser beam. For example, points with a corresponding laser beam angle difference less than or equal to a second predetermined threshold are classified as points corresponding to the same laser beam. In this way, the points in the point cloud data can be divided into point cloud groups corresponding one-to-one to each laser beam line of the LiDAR. Figure 6 Schematic diagram showing the angle of the laser beam at a point in the point cloud. Figure 6 As shown, for any two points in the current point cloud data: point P1 and point P2, the angle a1 between point P1 and the xy plane (i.e., the horizontal plane) can be calculated based on the three-dimensional coordinates of point P1 in the lidar coordinate system (xyz). The angle a1 is the angle of the laser beam corresponding to point P1. Similarly, the angle a2 between point P2 and the xy plane (i.e., the horizontal plane) can be calculated based on the three-dimensional coordinates of point P2 in the lidar coordinate system (xyz). The angle a2 is the angle of the laser beam corresponding to point P2. If the difference between a1 and a2 is less than or equal to the second predetermined threshold, it can be considered that point P1 and point P2 correspond to the same laser beam L1. If the difference between a1 and a2 is greater than the second predetermined threshold, then point P1 and point P2 correspond to different laser beams.

[0106] If the information of the point in the point cloud collected by the LiDAR includes not only the three-dimensional coordinates of the point (i.e., the LiDAR coordinate system coordinates) and the laser reflection intensity (intensity), but also the line value of the point, the LiDAR coordinate system coordinates of the second feature point can be directly determined using the first method mentioned above. If the information of the point in the point cloud collected by the LiDAR does not include the line value, the LiDAR coordinate system coordinates of the second feature point can be determined using the second method mentioned above. Of course, the two calculation methods can also be combined to better determine the point cloud points of the same laser beam.

[0107] Step S12, identifying edge feature points from the point cloud points corresponding to the same laser beam;

[0108] Specifically, the points where the distance in the neighborhood of each point cloud group suddenly changes are calculated. These points are edge feature points. Here, the edge feature points can include the feature points of the outer contour of the calibration plate itself and the outer contour feature points of each calibration hole. The first reference point of the calibration plate must be included in the outer contour feature points of the calibration plate itself, and the second reference point of the calibration plate can be determined based on the outer contour feature points of the calibration hole. It is known that the point cloud point has a distance value, so the distance in the field suddenly changes. For example, the difference between the distance value of the point and the distance value of the points within the predetermined distance range is greater than or equal to the fourth predetermined threshold.

[0109] Step S13: determining a second feature point based on the edge feature point.

[0110] In one implementation, the second reference point can be determined by performing shape fitting on the edge feature points of the calibration plate, performing shape fitting on the edge feature points to determine the second reference candidate point of the calibration hole, and selecting the second reference point from the second reference candidate points based on the attribute information of the calibration plate.

[0111] For example, if the calibration hole is circular, the second reference point of the calibration hole is the center of the circle. Shape fitting can be performed according to the circular model using, for example, an iterative algorithm based on random sampling (RANSAC, Random Sample Consensus), and the center point of the fitted shape is used as the second reference candidate point.

[0112] Specifically, still taking circular calibration holes as an example, the attribute information of the calibration plate includes the distance from the calibration hole to the edge of the calibration plate and the diameter of each calibration hole. The diameter of the circular shape corresponding to each second reference candidate point and the distance from the circular shape to the edge of the calibration plate are determined. If the diameter of the circular shape corresponding to a second reference candidate point is consistent with the diameter of the calibration hole, or the difference is within a predetermined range, and the distance from the edge of the circular shape corresponding to the second reference candidate point to the edge of the calibration plate is consistent with the distance from any calibration hole in the attribute information to the edge of the calibration plate, or the difference is within a predetermined range, the second reference candidate point can be considered a second reference point. Otherwise, the second reference candidate point is not a second reference point.

[0113] The present invention can determine the second reference candidate point by shape fitting of the calibration hole, and then verify the second reference candidate point with the attribute information of the calibration plate as the prior size constraint, and select the second reference candidate point that best meets the prior size from all the second reference candidate points as the second reference point. In this way, the second reference point can be accurately and efficiently extracted while reducing the computational complexity and hardware cost.

[0114] In one implementation, the first reference point can be determined by performing straight line fitting on the edge feature points to obtain at least two straight lines, determining the intersection of the at least two straight lines to obtain a first reference candidate point of the outer contour of the calibration plate, and determining the first reference point from the first reference candidate points based on the attribute information of the calibration plate.

[0115] In one example, RANSAC can be used to perform straight line fitting on edge feature points to obtain all straight lines in the edge feature points, and the three-dimensional coordinates of the intersection points of each straight line (i.e., the coordinates of the lidar coordinate system) are obtained and these intersection points are used as first reference candidate points. The first reference point is determined from the first reference candidate points based on the attribute information of the calibration plate.

[0116] In one example, the attribute information of the calibration plate may include the outer dimensions of the calibration plate. If the distances between n adjacent first reference candidate points are consistent with the outer dimensions of the calibration plate or the difference is within a predetermined range, then these n adjacent first reference candidate points are determined as first reference points. Here, n is the total number of first reference points on the single calibration plate. For example, if the calibration plate is a quadrilateral, its four corner points are selected as the first reference points of the calibration plate, and n is 4.

[0117] Taking the quadrilateral calibration plate as an example, if any four first reference candidate points satisfy "the distance between each first reference candidate point and its two adjacent first reference candidate points is consistent with the length and width of the calibration plate respectively", it indicates that the four adjacent first reference candidate points are likely to be the four corner points of the calibration plate, and the four first reference candidate points are determined as the first reference points. Figure 7 An example diagram showing the selection of a first reference point from first reference candidate points is shown. Figure 7 The four first reference candidate points p1, p2, p3, and p4 meet the above-mentioned requirement that "the distance between each first reference candidate point and its two adjacent first reference candidate points is consistent with the length and width of the calibration plate respectively". Therefore, the four first reference candidate points p1 to p4 are determined as the first reference points.

[0118] As can be seen from the above, the present disclosure determines the first reference point by fitting a straight line in the edge feature points, taking the intersection of the straight lines as the first reference candidate point, and then using the attribute information of the calibration plate to verify the first reference candidate point. The calculation complexity is low and the accuracy of feature point selection can be effectively improved.

[0119] Based on the above, the present disclosure designs two different second feature point extraction methods for different characteristics of lidar scanning methods and point cloud characteristics, which can adapt to the actual conditions of various types of lidars in actual application scenarios such as autonomous driving.

[0120] In step 207, an association can be established between the calibration plate coordinate system coordinates and the laser radar coordinate system coordinates of the second feature point and an ICP problem can be constructed to solve the first transformation relationship between the calibration plate coordinate system and the laser radar coordinate system.

[0121] In one example, the specific implementation process of step 207 may include the following steps S21 to S23:

[0122] Step S21, reading the coordinates of the first feature point of the calibration plate in the calibration plate coordinate system according to the plate identification of the calibration plate;

[0123] Step S22, using the coordinates of the calibration plate coordinate system of the first feature point as source point data and the coordinates of the laser radar coordinate system of the first feature point as target point data, constructing an ICP problem to obtain the current value of the first transformation relationship;

[0124] Specifically, according to the plate identification and marker identification, the calibration plate coordinate system coordinates and the laser radar coordinate system coordinates of the first feature point are sequentially read in the order in which they are stored in the structure object of the calibration plate, thereby obtaining the source data, target data, and their point correspondence relationship for constructing the ICP problem. The calibration plate coordinate system coordinates of the first feature point are used as the source data, and the laser radar coordinate system coordinates of the first feature point are used as the target data. Then, based on the preset initial value of the first transformation relationship, the projection data of the source data in the laser radar coordinate system is calculated, and the residual sum between the target data and the projection data is calculated. The residual sum is minimized by using the least squares method to optimize the first transformation relationship to obtain the current value of the first transformation relationship. In this implementation, while reading the calibration plate coordinate system coordinates and the laser radar coordinate system coordinates of the first feature point, their point correspondence relationship can be determined according to the plate identification, marker identification, and the default storage order. The ICP problem can be directly constructed without executing algorithms such as finding the nearest neighbor to determine the point correspondence relationship, thereby reducing the amount of calculation and the complexity of the calculation, further saving computing resources, and improving calibration efficiency.

[0125] The first transformation relationship may include a translation vector and a conversion matrix.

[0126] Step S23: Check whether the preset convergence condition is met. If so, the current value of the first transformation relationship is used as the final first transformation relationship. Otherwise, return to step S22 and continue iterating until the convergence condition is met.

[0127] The convergence condition can be pre-set as needed. The convergence condition can be, but is not limited to, the number of iterations reaching an upper limit, etc. The present disclosure does not limit the specific content of the convergence condition.

[0128] In step 209, the calibration plate coordinate system coordinates of the first feature point can be read from the structural object of the calibration plate according to the plate identification of the calibration plate, and the calibration plate coordinate system coordinates of the first feature point can be converted to the lidar coordinate system based on the first transformation relationship obtained in step 207 to obtain the lidar coordinate system coordinates of the first feature point.

[0129] The calibration in step 211 may include, but is not limited to, calibration of camera intrinsic parameters, calibration of camera extrinsic parameters, and joint calibration of the camera and LiDAR. Thus, step 211 can obtain at least one of the following parameters: camera intrinsic parameters, camera extrinsic parameters, and the transformation relationship between the camera coordinate system and the LiDAR coordinate system. The camera intrinsic parameters, camera extrinsic parameters, and the transformation relationship between the camera coordinate system and the LiDAR coordinate system can be used to unify the coordinates of the camera and LiDAR, and to fuse the data of the camera and LiDAR.

[0130] For example, in scenarios such as autonomous driving, assisted driving, and mobile robots, the camera intrinsic parameters, camera extrinsic parameters, and environmental images captured by multiple cameras can be used to estimate the position of obstacles within the camera's field of view, thereby realizing obstacle detection based on visual sensors.

[0131] For example, in applications such as simultaneous localization and mapping (SLAM), the camera intrinsic parameters, camera extrinsic parameters of full-view cameras and the estimation of environmental images collected by these cameras can be used to achieve high-definition map reconstruction and real-time positioning based on visual sensors.

[0132] For example, in a multi-sensor fusion scenario, the camera's intrinsic parameters, camera's extrinsic parameters, and the transformation relationship between the camera coordinate system and the lidar coordinate system can be used to achieve camera and lidar registration and data fusion, thereby obtaining more accurate and complete external environment information, and then achieving high-precision obstacle detection, path planning, and driving behavior decision-making, thereby improving driving safety under ADAS.

[0133] Of course, the parameter calibration of the embodiment of the present disclosure is not limited to the above three cases, and the application of the calibrated parameters is not limited to the above three examples.

[0134] In one implementation, if the camera intrinsic parameters are unknown or the camera intrinsic parameters are not accurate enough, the calibration in step 211 can be performed by Figure 8 The process shown is implemented as follows, see Figure 8 , the process includes the following steps:

[0135] Step 801: construct a PnP problem based on the laser radar coordinate system and the pixel coordinate system coordinates of the first feature point, and obtain the initial values ​​of the camera's intrinsic parameters and extrinsic parameters by optimizing and solving the PnP problem;

[0136] The pixel coordinates of each first feature point and the LiDAR coordinates of that first feature point can be read from the calibration plate's structure object in the order they are stored, forming point pair data. Each point pair contains both the LiDAR coordinates and the pixel coordinates of the same first feature point. Point-to-point (PnP) problems can be directly constructed using these point pairs. This significantly reduces the complexity and computational effort of constructing the PnP problem.

[0137] In one implementation, before constructing the PnP problem, the corresponding camera intrinsic parameter model and the PnP problem cost function can be determined based on the camera type. For example, a first PnP problem cost function corresponding to a long / short focus camera and a second PnP problem cost function corresponding to a fisheye camera can be preconfigured. For the camera intrinsic parameter model, a long / short focus camera can use the Pinhole Camera Model, while a fisheye camera can use the Scaramuzza Camera Model or the Double Sphere Camera Model. Thus, different PnP cost function interfaces can be provided for different camera types, enabling the calibration method of the disclosed embodiment to implement calibration of different types of lidars and cameras commonly used in scenarios such as autonomous driving.

[0138] In step 801, a corresponding camera intrinsic parameter model can be selected based on the camera type of the current camera to construct the projection process from the lidar coordinate system to the pixel coordinate system. For example, a pinhole camera model is used for a long / short focus camera, and a scaramuzza camera model or a double sphere camera model is used for a fisheye camera. A corresponding PnP problem cost function is selected based on the camera type of the current camera to construct the corresponding PnP problem cost function. For example, a first PnP problem cost function may be used for a long / short focus camera, while a second PnP problem cost function may be used for a fisheye camera. Thus, the PnP problem can be constructed using the camera intrinsic parameter model and PnP problem cost function adapted for the current camera, further improving the accuracy, versatility, and robustness of the calibration.

[0139] In step 801, the theoretical value of the intrinsic parameter and the manually measured value of the extrinsic parameter can be used as initial values ​​to perform an optimization solution to the PnP problem, and obtain the initial values ​​of the camera intrinsic parameter and the camera extrinsic parameter.

[0140] If m frames of image and m frames of point cloud data are obtained in step 201, where m is an integer greater than 1, m sets of pixel coordinate system coordinates of the first feature point can be obtained in step 203, and m sets of lidar coordinate system coordinates of the first feature point can be obtained in step 209. In this case, the theoretical values ​​of the intrinsic parameters and the manually measured values ​​of the extrinsic parameters can be used as initial values. A PnP problem can be constructed based on the m sets of pixel coordinate system coordinates and the m sets of lidar coordinate system coordinates of the first feature point, respectively. By optimizing and solving the PnP problem, m first intrinsic parameters and m first extrinsic parameters of the camera are obtained. The second intrinsic parameter of the camera is then determined based on the m first intrinsic parameters and used as the initial value of the camera intrinsic parameter. The m first extrinsic parameters are also used as the initial value of the camera extrinsic parameter.

[0141] There are various ways to determine the camera's second internal parameter based on the m first internal parameters. In one example, the average of the m first internal parameters can be used as the camera's second internal parameter. In another example, the weighted average of the m first internal parameters can be used as the camera's second internal parameter. Of course, any other applicable methods may also be used, and this disclosure does not limit this.

[0142] The specific method for determining the initial values ​​of the camera internal and external parameters disclosed in the present invention is described in detail below with reference to two different scenarios.

[0143] In a calibration scenario using a moving calibration plate, step 201 can obtain image sequences and point cloud sequences at n locations. Assuming that each image sequence contains m frames of images and each point cloud sequence contains m frames of point cloud data, this is equivalent to step 201 obtaining n*m frames of image and point cloud data. m sets of pixel coordinate system coordinates of first feature points will be obtained, each set containing pixel coordinate system coordinates of first feature points at n locations. A Point-N-Pair Problem (PnP) is constructed and solved for each set of pixel coordinate system coordinates and the corresponding LiDAR coordinate system coordinates. m second camera intrinsic parameters can be obtained, and the average of these m second camera intrinsic parameters is used as the first camera intrinsic parameter. When constructing the PnP problem for each set of pixel coordinate system coordinates, assuming that there are k first feature points on the calibration plate (k is an integer greater than 1), this is equivalent to constructing and solving the PnP problem using the pixel coordinate system coordinates of the n*k first feature points and their calibration plate coordinate system coordinates.

[0144] In the calibration scenario using the calibration room, assuming that there are n calibration plates in the calibration room, step 201 can obtain the image sequence and point cloud sequence of the n calibration plates in the calibration room. Assuming that the image sequence contains m frames of images and the point cloud sequence contains m frames of point cloud data, step 203 can obtain m groups of pixel coordinate system coordinates, each group of pixel coordinate system coordinates contains the pixel coordinate system coordinates of the first feature points of the n calibration plates. For each group of pixel coordinate system coordinates, a PnP problem is constructed based on the pixel coordinate system coordinates of the first feature point of each calibration plate and the corresponding lidar coordinate system coordinates, and the PnP problem of the n calibration plates in the group is optimized and solved. Similarly, m second camera intrinsic parameters can be obtained, and the average value of the m second camera intrinsic parameters is used as the first camera intrinsic parameter.

[0145] Step 803: calibrate the camera and the lidar based on the initial values ​​of the camera's internal parameters and external parameters.

[0146] In one specific embodiment, after step 801 or during step 803, the following may be included: optimizing the initial camera intrinsic parameter values ​​using the calibration plate coordinate system and pixel coordinate system coordinates of the first feature point. Specifically, a Point-to-Point (PnP) problem can be constructed based on the calibration plate coordinate system and pixel coordinate system coordinates of the first feature point, and the PnP problem can be optimized and solved based on the initial camera intrinsic parameter values ​​to obtain optimized camera intrinsic parameter values. In one example, the initial camera intrinsic parameter values ​​here can be the second intrinsic parameter described above. Thus, the calibration method disclosed herein can obtain camera intrinsic parameters with higher accuracy, better robustness, and better versatility.

[0147] Still taking the case of m frames of images and m frames of point cloud data as an example, in step 203, m groups of pixel coordinate system coordinates of the first feature point can be obtained. At this time, constructing a PnP problem based on the calibration plate coordinate system coordinates and the pixel coordinate system coordinates of the first feature point can include: constructing m PnP problems based on the m groups of pixel coordinate system coordinates of the first feature point and the calibration plate coordinate system coordinates of the first feature point, optimizing and solving the m PnP problems based on the initial values ​​of the camera's intrinsic parameters, obtaining m first intrinsic parameter optimization values ​​of the camera, and then determining the second intrinsic parameter optimization value of the camera based on the m first intrinsic parameter optimization values.

[0148] Similar to the second internal parameter described above, there are also various ways to determine the second internal parameter optimization value. For example, the average value, weighted average value, etc. of the m first internal parameter optimization values ​​can be taken as the second internal parameter optimization value. This disclosure does not limit the specific method for determining the second internal parameter optimization value.

[0149] For each set of pixel coordinate system coordinates, the PnP problem can be constructed in the following way: according to the plate identification and marking identification of the calibration plate, the calibration plate coordinate system coordinates and the pixel coordinate system coordinates of each first feature point are read in sequence according to the position order of the first feature point, and the calibration plate coordinate system coordinates and the pixel coordinate system coordinates corresponding to the same first feature point are combined to form a point pair. Based on the point pair, the camera intrinsic parameter model corresponding to the current camera and the PnP problem cost function corresponding to the current camera, the PnP problem is constructed according to the above formula (2).

[0150] In one example, the exemplary implementation process of calibrating the camera and the lidar based on the initial values ​​of the camera's internal parameters and external parameters in step 803 is shown in Figure 9 , which may include the following steps:

[0151] In step 901, a point-n-place (PnP) problem is constructed based on the coordinates of the first feature point in the calibration plate coordinate system and the pixel coordinate system. The PnP problem is then optimized and solved based on the initial values ​​of the camera's intrinsic parameters to obtain a second transformation relationship between the calibration plate coordinate system and the camera coordinate system. The initial values ​​of the camera's intrinsic parameters can be, but are not limited to, theoretical values ​​of the camera's intrinsic parameters, the second intrinsic parameter described above, or other values.

[0152] This step is identical to the aforementioned camera intrinsic parameter optimization steps. The details are discussed above and will not be repeated here. Using the aforementioned example of m frames of image and m frames of point cloud data, since m sets of pixel coordinates for the first feature point were obtained in step 203, m second transformation relationships can be obtained in step 901.

[0153] Step 903, determining the camera coordinate system coordinates of the second feature point according to the calibration plate coordinate system coordinates of the second feature point and the second transformation relationship;

[0154] Still taking the above case of m frames of images and m frames of point cloud data as an example, step 901 obtains m second transformation relationships, where m groups of camera coordinate system coordinates of the second feature points can be obtained.

[0155] Step 905: construct an ICP problem based on the camera coordinate system coordinates and the lidar coordinate system coordinates of the second feature point, optimize and solve the ICP problem based on the initial values ​​of the camera's extrinsic parameters, and obtain a third transformation relationship between the camera coordinate system and the lidar coordinate system.

[0156] For each set of camera coordinate system coordinates of the second feature point, the ICP problem can be constructed in the following manner: the lidar coordinate system coordinates of each second feature point are read in sequence according to the plate identification and marking identification of the calibration plate in the order of the positions of the second feature points, the lidar coordinate system coordinates and the camera coordinate system coordinates corresponding to the same first feature point are combined to form a point pair, and based on the point pair and the initial value of the camera extrinsic parameter obtained in step 901 (for example, the first extrinsic parameter corresponding to the current set of camera coordinate system coordinates among the m first extrinsic parameters mentioned above), the ICP problem is constructed according to the above formula (1) and optimized and solved to obtain the third transformation relationship between the camera coordinate system and the lidar coordinate system.

[0157] Using the example of m frames of image and m frames of point cloud data, we can first obtain the third transformation relationship between the m camera coordinate systems and the lidar coordinate system. Based on these third transformation relationships, we can determine the final value of the third transformation relationship between the camera coordinate system and the lidar coordinate system. For example, we can take an average or weighted average.

[0158] Therefore, the embodiment of the present disclosure first uses a camera intrinsic parameter model and a PnP problem cost function adapted to the current camera and lidar to determine the initial values ​​of the camera intrinsic parameters and the initial values ​​of the extrinsic parameters through the lidar coordinate system coordinates and the pixel coordinate system coordinates of the first feature point, and then calibrates the camera and lidar based on the initial values ​​of the intrinsic parameters and the initial values ​​of the extrinsic parameters. This not only can automatically complete the joint calibration of the internal and external parameters of the camera and lidar, but also can significantly improve the accuracy, robustness and versatility of the calibration results.

[0159] In one implementation, the calibration of step 211 can be performed by Figure 10 The process shown is implemented. Figure 10 As shown, the process may include the following steps:

[0160] Step 1001: Obtain a second transformation relationship between the calibration plate coordinate system and the camera coordinate system based on the calibration plate coordinate system coordinates and the pixel coordinate system coordinates of the first feature point;

[0161] In one example, a Point-to-Point (PnP) problem is constructed based on the coordinates of the calibration plate coordinate system and the pixel coordinate system of the first feature point. Based on the camera intrinsic parameters, the PnP problem is optimized to obtain the second transformation relationship between the calibration plate coordinate system and the camera coordinate system. Here, if the camera intrinsic parameters are known and relatively accurate and do not require further optimization, the camera intrinsic parameters can be kept unchanged during the PnP problem solution, and only the second transformation relationship between the calibration plate coordinate system and the camera coordinate system can be optimized until convergence conditions are met.

[0162] Still taking the case of m frames of images and m frames of point cloud data as an example, this step can obtain the second transformation relationship between the m calibration plate coordinate systems and the camera coordinate system.

[0163] Step 1003, obtaining the camera coordinate system coordinates of the second feature point according to the calibration plate coordinate system coordinates of the second feature point and the second transformation relationship;

[0164] Still taking the case of m frames of images and m frames of point cloud data as an example, if step 1001 obtains the second transformation relationship between the m calibration plate coordinate systems and the camera coordinate system, m groups of camera coordinate system coordinates of the second feature points will be obtained here.

[0165] Step 1005: Obtain a third transformation relationship between the camera coordinate system and the lidar coordinate system based on the camera coordinate system coordinates and the lidar coordinate system coordinates of the second feature point.

[0166] Still taking the case of m frames of images and m frames of point cloud data as an example, if m sets of camera coordinate system coordinates and m sets of lidar coordinate system coordinates of the second feature point are obtained before, an ICP problem can be constructed based on each set of camera coordinate system coordinates of the second feature point and its corresponding set of lidar coordinate system coordinates, and the ICP problem can be optimized and solved to obtain the third transformation relationship between the camera coordinate system and the lidar coordinate system.

[0167] If the third transformation relationship between m camera coordinate systems and the lidar coordinate system is obtained, the average value, weighted average value or other value can be taken as the final value of the third transformation relationship between the camera coordinate system and the lidar coordinate system.

[0168] Therefore, the external parameter calibration of the camera and lidar can be achieved by using the coordinates of the calibration plate coordinate system.

[0169] In some embodiments, the disclosed embodiments may also establish a marker coordinate system for each marker on the calibration plate, and store the marker coordinate system coordinates of the first feature point of each marker in sequence, and store them in association with the corresponding feature point sequence number, the marker identifier of the marker to which it belongs, and the plate identifier. The marker coordinate system is a three-dimensional rectangular coordinate system with a specific point in the marker (the center point or a vertex of the marker) as the origin. In some embodiments, the plane formed by the horizontal axis x and the vertical axis y of the three-dimensional rectangular coordinate system is the plane where the marker is located, and the z-axis of the three-dimensional rectangular coordinate system is perpendicular to the plane where the marker is located.

[0170] like Figure 11 As shown, the calibration method provided in the embodiment of the present disclosure may further include the steps of:

[0171] Step 1101: Acquire at least one frame of image of a calibration plate captured by a camera;

[0172] Step 1103: obtaining the marker coordinate system coordinates of each first feature point according to the attribute information of each marker on the calibration plate;

[0173] Step 1105: for each current frame image of at least one frame image, determining the pixel coordinate system coordinates of the first feature point from the current frame image;

[0174] Step 1107: Obtaining a transformation relationship from the marker coordinate system of each marker on the calibration plate to the camera coordinate system according to the marker coordinate system coordinates and the pixel coordinate system coordinates of each first feature point on the calibration plate; and

[0175] Step 1109: Obtain the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate according to the transformation relationship from the marker coordinate system of each marker on the calibration plate to the camera coordinate system.

[0176] In one implementation, step 1101 includes:

[0177] Step 1101A: Acquire an image sequence captured by a camera;

[0178] Step 1101B: perform image detection on each frame of the image sequence to determine at least one frame of the image that contains a calibration plate and the number of marks in the calibration plate is greater than or equal to a predetermined threshold.

[0179] In one implementation, the transformation relationship from the calibration plate coordinate system to the camera coordinate system of the calibration plate in step 1109 is the average value or weighted average value of the transformation relationships from the marker coordinate systems of the markers on the calibration plate to the camera coordinate system.

[0180] In another implementation, step 1109 includes:

[0181] Step 1107A: Obtaining an initial value of the transformation relationship between the calibration plate coordinate system and the camera coordinate system according to the transformation relationship between the marker coordinate system of each marker on the calibration plate and the camera coordinate system;

[0182] Step 1107B: Based on the initial value of the transformation relationship, the optimized value of the transformation relationship from the calibration plate coordinate system to the camera coordinate system corresponding to the current frame image is obtained through optimization solution.

[0183] In one implementation, the pixel coordinate system coordinates in step 1105 are detection coordinates in the pixel coordinate system, then Figure 11 The calibration method shown may also include:

[0184] Step 1111: Based on the transformation relationship from the calibration plate coordinate system to the camera coordinate system and the attribute information of each marker on the calibration plate, the projection coordinates of each first feature point on the calibration plate in the pixel coordinate system are obtained. The transformation relationship from the calibration plate coordinate system to the camera coordinate system can be an initial value of the transformation relationship or an optimized value of the transformation relationship.

[0185] Step 1113: Determine a reprojection error of the calibration plate based on the detected coordinates and projected coordinates of each first feature point on the calibration plate in the pixel coordinate system. The reprojection error of the calibration plate is the average or weighted average of the reprojection errors of each first feature point on the calibration plate.

[0186] Step 1115: determining, from at least one frame of image captured by the camera, a plurality of target images whose reprojection error of the calibration plate is less than or equal to a predetermined threshold;

[0187] Step 1117: Based on the first camera coordinate system coordinates of each feature point in the multi-frame target image, the second camera coordinate system coordinates of each feature point on the calibration plate are obtained. Each feature point includes a first feature point of the marker and a second feature point of the structural outline. The second coordinates are the average or weighted average of the first coordinates in the multi-frame target image. Thus, the second camera coordinate system coordinates are more accurate than the first camera coordinate system coordinates. The second camera coordinate system coordinates of each feature point can be correlated with other coordinates, such as the calibration plate coordinate system, the LiDAR coordinate system, or the calibration inter-coordinate system, to obtain the corresponding transformation relationship.

[0188] A variety of predetermined thresholds have been mentioned above. Those skilled in the art may set the values ​​of the predetermined thresholds corresponding to the various parameters as needed, and this disclosure does not impose any restrictions thereon. Furthermore, when performing a weighted average on multiple coordinates, weights may be assigned based on the magnitude of the reprojection error of each feature point, with larger reprojection errors receiving lower weights and smaller reprojection errors receiving higher weights.

[0189] In one implementation, step 1111 includes:

[0190] Step 1111A: Determine the camera projection model (e.g., the pinhole model used by a short / telephoto camera, the Scaramuzza model used by a fisheye camera, etc.) based on the camera type information.

[0191] Step 1111B: Calculate the camera coordinate system coordinates of each first feature point on the calibration plate according to the optimized value of the transformation relationship and the calibration plate coordinate system coordinates of each first feature point;

[0192] Step 1111C: Based on the camera projection model and the camera coordinate system coordinates of each first feature point on the calibration plate, calculate the projection coordinates of each first feature point on the calibration plate in the pixel coordinate system.

[0193] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0194] Exemplary devices

[0195] According to an embodiment of another aspect, a calibration device is provided. Figure 12 FIG. 1 is a schematic block diagram of a calibration device in the form of a computer program according to an embodiment. Figure 12 As shown, the apparatus 1200 includes:

[0196] An acquisition unit 1201 is configured to acquire images and point cloud data of a calibration plate collected by a camera and a lidar, respectively, wherein the calibration plate is provided with a marker, and the calibration plate includes first feature points of the marker and second feature points of a structural profile of the calibration plate;

[0197] An image detection unit 1202 is configured to determine pixel coordinates of a first feature point based on the image;

[0198] The point cloud extraction unit 1203 is used to determine the laser radar coordinate system coordinates of the second feature point based on the point cloud data;

[0199] a transformation relationship determining unit 1204, configured to determine a first transformation relationship between the calibration plate coordinate system and the lidar coordinate system based on the calibration plate coordinate system coordinates and the lidar coordinate system coordinates of the second feature point;

[0200] A coordinate conversion unit 1205 is configured to determine the laser radar coordinate system coordinates of the first feature point according to the calibration plate coordinate system coordinates of the first feature point and the first transformation relationship; and

[0201] The joint calibration unit 1206 is used to calibrate the parameters of the camera and the lidar according to the pixel coordinate system coordinates and the lidar coordinate system coordinates of the first feature point.

[0202] The specific details of the functions of the above-mentioned components can be found in the previous method section and will not be repeated here.

[0203] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0204] Exemplary Storage Media

[0205] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The program includes instructions, and when the instructions are executed by one or more processors of a computing device, the steps of any one of the method embodiments described above are executed.

[0206] Exemplary electronic devices

[0207] The calibration device provided in the embodiment of the present disclosure may be implemented in the form of an electronic device. In this case, Figure 13 As shown in , the calibration device includes one or more processors 1301, and also includes a memory 1302 for storing one or more programs, which are executed by the one or more processors 1301 to implement the method flow shown in the embodiment of the present disclosure and / or the program units corresponding to each unit in the device.

[0208] The various components are interconnected using different buses and can be mounted on a common motherboard or in other ways as needed. The processor 1301 can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of the user interface on an external input / output device (such as a display device coupled to the interface). In other embodiments, if desired, multiple processors and / or multiple buses can be used with multiple memories and multiple memories.

[0209] The processor 1301 may include one or more single-core processors or multi-core processors. The processor 1301 may include any combination of general-purpose processors or specialized processors (such as an image processor, an application processor, a baseband processor, etc.).

[0210] The memory 1302 is a computer-readable storage medium provided by the present disclosure, which can be used to store non-transient software programs, non-transient computer executable programs and units, such as the embodiment of the present disclosure. Figure 2 The processor 1301 executes the non-transient software programs, instructions and units stored in the memory 1302, thereby executing the above-mentioned method embodiment. Figure 2 The procedures, instructions and units corresponding to the calibration method shown.

[0211] The calibration device may further include: an input device 1303 and an output device 1304. The processor 1301, the memory 1302, the input device 1303 and the output device 1304 may be connected via a bus or other means. Figure 13 The bus connection is taken as an example.

[0212] The input device 1303 can receive input digital or character information, and generate signal input related to the user settings and function control of the calibration device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, an indicator stick, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1304 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen. The above-mentioned program (also referred to as software, software application, or code) includes machine instructions for a programmable processor, and these computer programs can be implemented using an object-oriented programming language, assembly or machine language.

[0213] As time goes by and technology develops, the meaning of medium becomes more and more extensive, and the propagation path of computer programs is no longer limited to tangible media, but can also be downloaded directly from the Internet, etc. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can adopt but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, device or device.

[0214] A9. A method as described in A8, wherein the calibration plate has a plate identification, a plurality of marks are provided on the calibration plate, and each mark has a corresponding mark identification, and the method further comprises: according to the position sequence of each mark in the calibration plate, sequentially associating and storing the calibration plate coordinate system coordinates of each first feature point in the mark with the corresponding mark identification and plate identification; according to the position sequence of each calibration hole in the calibration plate, sequentially associating and storing the calibration plate coordinate system coordinates of each second feature point in the calibration plate with the corresponding plate identification.

[0215] A10. A method as described in A1, wherein the first transformation relationship between the calibration plate coordinate system and the lidar coordinate system is determined based on the calibration plate coordinate system coordinates and the lidar coordinate system coordinates of the second feature point, including: establishing an association between the calibration plate coordinate system coordinates and the lidar coordinate system coordinates according to the second feature point and constructing an ICP problem to solve the first transformation relationship between the calibration plate coordinate system and the lidar coordinate system.

[0216] A11. The method as described in A1 is characterized in that the parameters of the camera and the lidar are calibrated according to the pixel coordinate system coordinates and the lidar coordinate system coordinates of the first feature point, including: constructing a PnP problem according to the lidar coordinate system coordinates and the pixel coordinate system coordinates of the first feature point, and obtaining the initial values ​​of the intrinsic parameters and the initial values ​​of the extrinsic parameters of the camera by optimizing and solving the PnP problem; and calibrating the camera and the lidar based on the initial values ​​of the intrinsic parameters and the initial values ​​of the extrinsic parameters of the camera.

[0217] A12. A method as described in A11, wherein the parameter calibration of the camera and the lidar based on the initial values ​​of the intrinsic parameters and the initial values ​​of the extrinsic parameters of the camera includes: constructing a PnP problem according to the coordinates of the calibration plate coordinate system and the pixel coordinate system of the first feature point, and optimizing and solving the PnP problem based on the initial values ​​of the intrinsic parameters of the camera to obtain a second transformation relationship between the calibration plate coordinate system and the camera coordinate system; determining the camera coordinate system coordinates of the second feature point according to the coordinates of the calibration plate coordinate system of the second feature point and the second transformation relationship; constructing an ICP problem according to the coordinates of the camera coordinate system and the lidar coordinate system of the second feature point, and optimizing and solving the ICP problem based on the initial values ​​of the extrinsic parameters of the camera to obtain a third transformation relationship between the camera coordinate system and the lidar coordinate system.

[0218] A13. The method as described in A11 further includes: determining a corresponding camera intrinsic parameter model and a cost function of the PnP problem according to the type of the camera.

[0219] A14. The method as described in A11 also includes: constructing a PnP problem based on the coordinates of the calibration plate coordinate system and the pixel coordinate system of the first feature point, optimizing and solving the PnP problem based on the initial value of the intrinsic parameter of the camera, and obtaining the optimized value of the intrinsic parameter of the camera.

[0220] A15. A method as described in A1, wherein the camera and lidar are calibrated according to the lidar coordinate system coordinates and the pixel coordinate system coordinates of the first feature point, including: obtaining a second transformation relationship between the calibration plate coordinate system and the camera coordinate system according to the calibration plate coordinate system coordinates and the pixel coordinate system coordinates of the first feature point; obtaining the camera coordinate system coordinates of the second feature point according to the calibration plate coordinate system coordinates and the second transformation relationship; and obtaining a third transformation relationship between the camera coordinate system and the lidar coordinate system according to the camera coordinate system coordinates and the lidar coordinate system coordinates of the second feature point.

[0221] A16. A method as described in A1 or A11, wherein the calibration plate has n positions, different positions among the n positions have different heights, angles and / or distances relative to the loading platforms of the camera and the lidar and are all located in the common viewing area of ​​the camera and the lidar, and n is an integer greater than 1; the image and point cloud data about the calibration plate include n image sequences and n point cloud sequences corresponding to the n positions, and the image frames in each image sequence and the point cloud frames in the point cloud sequence at the same position are at least partially acquired at the same time.

[0222] A17. A method as described in A1 or A11, wherein the image and point cloud data about the calibration plate are: an image sequence and a point cloud sequence about the calibration room where the calibration plate is located, a plurality of calibration plates are provided in the calibration room and the positions of the plurality of calibration plates are different, each image frame in the image sequence includes part or all of the calibration plates in the calibration room, each point cloud frame in the point cloud sequence includes point cloud data of part or all of the calibration plates in the calibration room, and the image frames in the image sequence and the point cloud frames in the point cloud sequence are at least partially acquired at the same time.

[0223] A18. A method as described in A16 or 17A, wherein the image sequence includes m frames of images, the point cloud sequence includes m frames of point cloud data, and m is an integer greater than 1; the laser radar coordinate system coordinates of the first feature point include: m groups of laser radar coordinate system coordinates obtained based on the m frames of images and point cloud data; the pixel coordinate system coordinates of the first feature point include: m groups of pixel coordinate system coordinates obtained based on the m frames of images; the initial value of the camera's intrinsic parameter is obtained by: constructing a PnP problem based on the m groups of laser radar coordinate system coordinates and m groups of pixel coordinate system coordinates of the first feature point, respectively, and obtaining m second intrinsic parameters of the camera by optimizing and solving the PnP problem; and determining the first intrinsic parameter of the camera based on the m second internal parameters of the camera, and using the first internal parameter of the camera as the initial value of the camera's intrinsic parameter.

[0224] The technical solutions provided by the present disclosure are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and core ideas of the present disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scopes based on the ideas of the present disclosure. In summary, the content of this specification should not be understood as a limitation on the present disclosure. The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A calibration method, characterized in that: The method comprises: Acquire images and point cloud data of a calibration plate collected by a camera and a laser radar, respectively, wherein the calibration plate is provided with a mark, and the calibration plate includes a first feature point of the mark and a second feature point of a structural contour of the calibration plate; determining pixel coordinates of a first feature point according to the image; Determine the laser radar coordinate system coordinates of the second feature point according to the point cloud data; Determining a first transformation relationship between the calibration plate coordinate system and the lidar coordinate system according to the calibration plate coordinate system coordinates and the lidar coordinate system coordinates of the second feature point; Determining the laser radar coordinate system coordinates of the first feature point according to the calibration plate coordinate system coordinates of the first feature point and the first transformation relationship; and The camera and the lidar are calibrated according to the pixel coordinate system coordinates and the lidar coordinate system coordinates of the first feature point.

2. The method according to claim 1, wherein The first feature point includes a corner point of the mark; the second feature point includes at least one of the following: a first reference point about the outer contour of the calibration plate and a second reference point about the calibration hole in the calibration plate.

3. The method according to claim 2, wherein: The calibration plate is provided with a plurality of the marks and a plurality of the calibration holes, each mark has a corresponding mark identification, the mark is a QR code, the first reference point is a corner point of the outer contour of the calibration plate, and the second reference point is the center point of the calibration hole.

4. The method according to any one of claims 1 to 3, wherein: The determining the laser radar coordinate system coordinates of the second feature point according to the point cloud data includes: identifying point cloud points corresponding to the same laser beam from the point cloud data; Identifying edge feature points from the point cloud points corresponding to the same laser beam; The second feature point is determined according to the edge feature point.

5. The method according to claim 4, wherein: Identifying point cloud points corresponding to the same laser beam from the point cloud data includes: Determining the point cloud points corresponding to the same laser beam according to the laser beam identifier of each point cloud point in the point cloud data; or The angle of the corresponding laser beam is determined according to the coordinates of each point cloud point in the point cloud data, and points whose angle differences of the corresponding laser beams are less than a predetermined threshold are determined as point cloud points corresponding to the same laser beam.

6. The method of claim 4, wherein: Determining the second feature point according to the edge feature point includes: Performing straight line fitting on the edge feature points to obtain at least two straight lines; determining an intersection point of the at least two straight lines to obtain a first candidate reference point about the outer contour of the calibration plate; and The first reference point is determined from the first reference candidate points based on the attribute information of the calibration plate.

7. The method of claim 4, wherein: Determining the second feature point based on the edge feature point includes: Performing shape fitting on the edge feature points to determine second reference candidate points about the calibration hole; The second reference point is selected from the second reference candidate points based on the attribute information of the calibration plate and the calibration hole.

8. The method according to claim 1, wherein Also includes: Establishing a calibration plate coordinate system and a structure object of the calibration plate; The calibration plate coordinate system coordinates of each first feature point and each second feature point on the calibration plate are stored in the structure object according to the attribute information of the calibration plate.

9. A calibration device, characterized in that: The device comprises: one or more processors, and A memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a program, the program comprising instructions which, when executed by one or more processors of a computing device, cause the computing device to perform the method according to any one of claims 1 to 8.

Images