Parameter calibration method for laser radar and camera

Abstract

The invention discloses a parameter calibration method for a laser radar and a camera. The method comprises the following steps: acquiring a scene point cloud and a scene image respectively acquired by the laser radar and the camera for the same scene; identifying edge point clouds in the scene point clouds; identifying edge pixel points in the scene image; according to a predetermined initial conversion relation, converting the coordinate value of the edge point cloud in the laser radar coordinate system into a coordinate value in a camera pixel coordinate system to obtain an edge projection point cloud; constructing a target function according to coordinate distance deviation between each projection point in the edge projection point cloud and each edge pixel point in a camera pixel coordinate system; and performing optimization iteration on each calibration parameter in the initial conversion relation to obtain a calibration parameter enabling the target function to be minimum, and taking the initial conversion relation determined by the calibration parameter enabling the target function to be minimum as the calibration conversion relation between the laser radar and the camera. According to the technical scheme, the calibration difficulty is reduced on the basis of ensuring the calibration accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of lidar scanning, and particularly to a method for calibrating parameters of a lidar and a camera. Background Art

[0002] A lidar can obtain rich three-dimensional information in a scene, and a camera can obtain rich color texture information; the fusion result of the two data applied to scenarios such as autonomous driving can greatly improve the accuracy and reliability of target detection. However, high-precision fusion of the data collected by the lidar and the camera respectively requires the precise relative pose relationship between the lidar and the camera. Therefore, how to achieve precise calibration of the relative pose relationship between the lidar and the camera is one of the key issues that need to be focused on in the industry. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for calibrating parameters of a lidar and a camera, which uses the edge information of the object scene in the environment for calibration, reduces the limitation on the calibration site, improves the calibration accuracy, and reduces the calibration difficulty.

[0004] To solve the above technical problems, the present invention provides a method for calibrating parameters of a lidar and a camera, including:

[0005] Obtain the scene point cloud and the scene image collected by the lidar and the camera for the same scene respectively;

[0006] Identify the edge point cloud in the scene point cloud; and identify the edge pixel points in the scene image;

[0007] According to the pre-determined initial conversion relationship, convert the coordinate values of the edge point cloud in the lidar coordinate system to the coordinate values in the camera pixel coordinate system to obtain the edge projected point cloud;

[0008] Construct an objective function according to the coordinate distance deviation between each edge projected point in the edge projected point cloud and each of the edge pixel points in the camera pixel coordinate system;

[0009] Optimize and iterate each calibration parameter in the initial conversion relationship to obtain the calibration parameters that minimize the objective function, and use the initial conversion relationship determined by the calibration parameters that minimize the objective function as the calibration conversion relationship between the lidar and the camera.

[0010] In an optional embodiment of the present application, identifying the edge point cloud in the scene point cloud includes:

[0011] According to the echo label corresponding to each sampling point in the scene point cloud, obtain the echo sequence number and the reflection intensity corresponding to each sampling point;

[0012] Identify the initial edge point cloud determined by the first echo in the double echo in the scene point cloud according to the echo serial number;

[0013] Remove the initial edge points in the initial edge point cloud whose reflection intensity is greater than the set intensity threshold according to the reflection intensity;

[0014] Screen the initial edge point cloud after removal to obtain the edge point cloud.

[0015] In an alternative embodiment of the present application, screening the initial edge point cloud after removal to obtain the edge point cloud includes:

[0016] Perform a neighboring point search on each initial edge point in the initial edge point cloud after removal to obtain m neighboring points corresponding to each initial edge point;

[0017] Perform a centroid point operation on the m neighboring points corresponding to each initial edge point to obtain a neighboring centroid corresponding to each initial edge point;

[0018] Perform an Euclidean distance operation on each initial edge point and the corresponding neighboring centroid to obtain a neighboring Euclidean distance corresponding to each initial edge point;

[0019] Screen out the initial edge points in the initial edge point cloud whose corresponding neighboring Euclidean distance is not less than the set distance threshold as the edge points in the obtained edge point cloud.

[0020] In an alternative embodiment of the present application, identifying the edge pixel points in the scene image includes:

[0021] Perform grayscale processing on the scene image to obtain a grayscale image;

[0022] Use the Canny operator to extract the edge pixel points in the grayscale image.

[0023] In an alternative embodiment of the present application, the process of pre-determining the initial conversion relationship includes:

[0024] Determine a set rotation matrix and an initial translation vector representing the conversion relationship between the lidar coordinate system and the camera coordinate system according to the relative position relationship between the lidar and the camera;

[0025] Convert the set rotation matrix into an initial rotation matrix representing each matrix element with quaternion parameters;

[0026] Use the initial rotation matrix and the initial translation vector together as the initial conversion relationship;

[0027] Correspondingly, according to a pre-determined initial conversion relationship and the internal camera parameters, the coordinate values of the edge point cloud in the lidar coordinate system are converted to the coordinate values in the camera pixel coordinate system to obtain an edge projection point cloud, including:

[0028] According to the initial conversion relationship, the coordinate values of the edge point cloud in the lidar coordinate system are converted to the camera coordinate system to obtain the coordinate values of each edge point of the edge point cloud in the camera coordinate system;

[0029] According to the internal camera parameters, the coordinate values of the edge point cloud in the camera coordinate system are converted to the camera pixel coordinate system to obtain the edge projection point cloud.

[0030] In an alternative embodiment of the present application, according to the internal camera parameters, converting the coordinate values of the edge point cloud in the camera coordinate system to the camera pixel coordinate system to obtain the edge projection point cloud includes:

[0031] Performing a normalized plane conversion on the coordinate values of the edge point cloud in the camera coordinate system, and performing a perspective conversion on the edge point cloud according to the coordinate values after the normalized conversion of the edge point cloud and the internal camera parameters to obtain the edge projection point cloud of the edge point cloud in the camera image coordinate system;

[0032] According to the abscissa range and ordinate range of the imaging plane photosensitive range of the camera in the camera image coordinate system, screening out the edge projection points in the edge point cloud whose abscissa is within the abscissa range and whose ordinate is within the ordinate range;

[0033] The coordinate values of the screened edge point cloud in the camera image coordinate system are discretely converted to the camera pixel coordinate system to obtain the pixel coordinate values of the edge projection point cloud in the pixel coordinate system.

[0034] In an alternative embodiment of the present application, optimizing and iterating each calibration parameter in the initial conversion relationship includes:

[0035] Optimizing and iterating each quaternion parameter in the initial conversion matrix and each translation parameter in the initial translation vector.

[0036] In an alternative embodiment of the present application, constructing an objective function according to the coordinate distance deviation between each edge projection point in the edge projection point cloud and each edge pixel point in the camera pixel coordinate system includes:

[0037] According to the coordinate values of each edge projection point in the edge projection point cloud in the camera pixel coordinate system, perform a neighboring point search for each edge projection point among the edge pixel points to obtain neighboring pixel points corresponding to each edge projection point one by one;

[0038] Perform an Euclidean distance operation between each edge projection point and the corresponding neighboring pixel point, and use the sum of the Euclidean distances between all the edge projection points and the corresponding neighboring pixel points as the objective function.

[0039] In an optional embodiment of the present application, performing a neighboring point search for each edge projection point among the edge pixel points to obtain neighboring pixel points corresponding to each edge point one by one includes:

[0040] Construct a KD-Tree for each edge pixel point, use each edge projection point as a search point, and find the edge pixel point closest to the search point as the neighboring pixel point corresponding to the search point.

[0041] In an optional embodiment of the present application, optimizing and iterating each calibration parameter in the initial conversion relationship includes:

[0042] Optimize and iterate each calibration parameter. When the norm of the change amount of the calibration parameter in two consecutive optimization iterations is lower than a preset threshold, or the number of iterations reaches a preset number, the iteration process ends.

[0043] A method for calibrating the parameters of a lidar and a camera provided by the present invention includes obtaining a scene point cloud and a scene image respectively collected by the lidar and the camera for the same scene; identifying edge point clouds in the scene point cloud; and identifying edge pixel points in the scene image; according to a pre-determined initial conversion relationship, converting the coordinate values of the edge point cloud in the lidar coordinate system to the coordinate values in the camera pixel coordinate system to obtain an edge projection point cloud; constructing an objective function according to the coordinate distance deviation between each projection point in the edge projection point cloud and each edge pixel point in the camera pixel coordinate system; optimizing and iterating each calibration parameter in the initial conversion relationship to obtain calibration parameters that minimize the objective function, and using the initial conversion relationship determined by the calibration parameters that minimize the objective function as the calibration conversion relationship between the lidar and the camera.

[0044] Based on the data acquisition of the same scene using lidar and camera in this application, edge point clouds in the scene point cloud collected by the lidar are screened out, and at the same time, edge pixel points with stronger identification information in the scene image collected by the camera are identified; obviously, the edge point clouds and edge pixel points should represent the data information of the same object edge contour; theoretically speaking, if the accurate relative pose relationship between the lidar and the camera, that is, the relative conversion relationship between the two coordinate systems, can be obtained, after the edge point clouds and edge pixel points are converted to the same coordinate system, the edge point clouds should be able to coincide with the edge pixel points; based on this, in this application, the relative deviation between the edge point clouds and edge pixel points after being converted to the same coordinate system is used as the objective function, and the calibration parameters representing the conversion relationship between the lidar and the camera are iteratively calculated repeatedly, and finally the calibration parameters that can minimize the objective function are determined, and the calibration conversion relationship between the lidar and the camera is determined accordingly. During the calibration process of the conversion relationship between the lidar and the camera in this application, only the data information of the object contour edge in the scene is required, which not only reduces the amount of data for calibration operations to a certain extent, but also does not increase the computational complexity in this application even in the face of extremely complex scenes, thus reducing the calibration difficulty on the basis of ensuring the calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0046] Figure 1 It is a schematic flowchart of the parameter calibration method for the lidar and the camera provided by the embodiment of this application;

[0047] Figure 2 It is a schematic diagram of the relative position relationship between the lidar coordinate system and the three coordinate systems of the camera provided by the embodiment of this application;

[0048] Figure 3 It is a schematic diagram of the double - return optical path of the lidar;

[0049] Figure 4 It is a schematic diagram comparing the double - return optical paths of the lidar in two different scenes provided by the embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] The core of the present invention is to provide a parameter calibration method for a lidar and a camera, which reduces the calibration difficulty on the basis of ensuring the calibration accuracy of the conversion relationship between the lidar and the camera.

[0051] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0052] As Figures 1 to 4 shown, Figure 1 is a schematic flowchart of the parameter calibration method for the lidar and the camera provided in the embodiment of the present application; Figure 2 is a schematic diagram of the relative position relationship between the lidar coordinate system and the three coordinate systems of the camera provided in the embodiment of the present application; Figure 3 is a schematic diagram of the double return optical path of the lidar; Figure 4 is a schematic diagram for comparing the double return optical paths of the lidar in two different scenarios provided in the embodiment of the present application.

[0053] To facilitate the understanding of the subsequent technical solutions, the lidar coordinate system and the camera coordinate system, camera image coordinate system, and camera pixel coordinate system involved in the camera will be briefly introduced below.

[0054] As Figure 2 shown, the camera coordinate system is also Figure 2 a three-dimensional rectangular coordinate system Oc-XcYcZc with the optical center Oc of the eyepiece in the camera as the origin and the direction of the eyepiece optical axis as the direction of the Zc axis; the camera image coordinate system is a two-dimensional rectangular coordinate system O1-xy on the imaging plane of the camera. Among them, the origin O1 of the camera image coordinate system is the center point of the imaging plane and is also located on the Zc axis. The x-axis and the y-axis are respectively parallel to the length direction and the width direction of the imaging plane; the camera pixel coordinate system is also a two-dimensional rectangular coordinate system O-uv on the imaging plane, and the origin O of the camera pixel coordinate system is the vertex of the imaging plane.

[0055] It can be understood that the conversion between the lidar coordinate system and the camera coordinate system in the present application is similar to the conversion between the world coordinate system and the camera coordinate system and belongs to a rigid conversion. In addition, based on the common knowledge of the conversion between the three different coordinate systems in the camera, the conversion between the camera coordinate system and the camera image coordinate system should be a perspective projection conversion, and the conversion between the camera image coordinate system and the camera pixel coordinate system should be a discretization conversion.

[0056] Based on the above discussion, in a specific embodiment of the present application, the parameter calibration method for the lidar and the camera may include:

[0057] S1: Obtain the scene point cloud and the scene image collected by the lidar and the camera respectively for the same scene.

[0058] S2: Identify the edge point cloud in the scene point cloud; and identify the edge pixel points in the scene image.

[0059] Based on the characteristics of the point cloud collected by the lidar's scanning, it is known that the lidar emits laser light towards the target object, and determines the position of the position points on the surface of the target object in space according to the light received from the surface of the target object reflected back. And as Figure 3 shown, when a beam of laser light irradiates the edge of the target object, the laser beam will be cut by the edge contour of the target object, so that part of the light is reflected from the edge of the target object, and the other part of the light is reflected by the subsequent object of the target object. Thus, the same beam of light emitted by the lidar can receive two echo signals reflected back successively, that is, the double echo signal; obviously, the sampling point determined by the first echo signal in the double echo signal is the position point on the edge contour of the target object; based on this, the edge point cloud in the scene point cloud can be identified and determined in this application.

[0060] For the identification of the edge pixel points in the scene image, conventional image recognition techniques can be directly used. For example, the scene image can be first grayscale processed, and then the Canny operator can be used to extract the edge pixel points in the grayscale image. Of course, other image recognition techniques are not excluded in this application for the identification of edge pixel points, and this is not listed one by one in this application.

[0061] It can be understood that the position information corresponding to the identified edge point cloud should be the coordinate values of each edge point cloud in the lidar coordinate system. And the position information of the edge pixel points is the coordinate values of each edge pixel point in the camera pixel coordinate system.

[0062] S3: According to the pre-determined initial conversion relationship and the camera internal parameters, convert the coordinate values of the edge point cloud in the lidar coordinate system to the coordinate values in the camera pixel coordinate system to obtain the edge projection point cloud.

[0063] The initial conversion relationship in this embodiment can represent the conversion relationship between the lidar coordinate system and the camera coordinate system. The initial conversion relationship in this embodiment can be set based on the relative position relationship between the lidar and the camera. For the integrated device of the lidar and the camera, the initial conversion relationship can be the conversion relationship provided by the manufacturer when the integrated device leaves the factory, and no specific limitation is made in this application.

[0064] Based on Figure 2 the conversion relationship between the lidar coordinate system and the camera coordinate system shown can adopt a rotation matrix and a translation vector It is represented by. In the subsequent process of optimizing and iterating the conversion relationship, the elements and parameters in the rotation matrix and translation vector that need to be optimized and adjusted are precisely those, until a set of element parameters that can accurately represent the conversion relationship between the lidar and the camera is obtained.

[0065] However, further considering that there are certain limitations in the rotation matrix during iterative operations, for this reason, the process of determining the initial conversion relationship between the lidar coordinate system and the camera coordinate system in this embodiment can further include:

[0066] According to the relative position relationship between the lidar and the camera, determine the set rotation matrix and initial translation vector that represent the conversion relationship between the lidar coordinate system and the camera coordinate system;

[0067] Convert the set rotation matrix into an initial rotation matrix representing each matrix element with quaternion parameters;

[0068] Use the initial rotation matrix and the initial translation vector together as the initial conversion relationship.

[0069] Furthermore, the process of converting the set rotation matrix into an initial rotation matrix represented by quaternion parameters in this embodiment can be as follows:

[0070] 1) Convert the set rotation matrix into the Euler angle formula: ; where are respectively the yaw angle, pitch angle, and roll angle when the three coordinate axes of the first three-dimensional rectangular coordinate system rotation are parallel to the three coordinate axes of the second three-dimensional rectangular coordinate system.

[0071] 2) Convert the Euler angle formula into the quaternion formula:

[0072] ;

[0073] where .

[0074] 3) Convert the quaternion formula into an initial rotation matrix representing each matrix element with quaternion parameters:

[0075] .

[0076] Thus, the initial rotation matrix and the initial translation vector can jointly form the initial conversion relationship; and in the subsequent process of optimizing and iterating the initial conversion relationship, it is to optimize and iterate each quaternion parameter in the initial conversion matrix and each translation parameter in the initial translation vector.

[0077] Since the coordinate values of the edge point cloud in the lidar coordinate system need to be finally converted to the camera pixel coordinate system, but the initial conversion relationship can only convert the coordinate values of the edge point cloud in the lidar coordinate system to the camera coordinate values. Therefore, after obtaining the coordinate values of the edge point cloud in the camera coordinate system, the coordinate values of the edge point cloud in the camera coordinate system should be further converted to the camera pixel coordinate system according to the internal parameters of the camera, and the edge projection point cloud can be obtained.

[0078] Furthermore, the process of converting the coordinate values of the edge point cloud in the camera coordinate system to the camera pixel coordinate system may include:

[0079] S31: Perform a normalized plane conversion on the coordinate values of the edge point cloud in the camera coordinate system to obtain the coordinate values of the edge point cloud in the camera image coordinate system;

[0080] S32: Determine the horizontal coordinate range and vertical coordinate range that can be imaged on the imaging plane of the camera in the camera image coordinate system according to the internal parameters of the camera;

[0081] S33: Filter out the edge points in the edge point cloud whose horizontal coordinates are within the horizontal coordinate range and vertical coordinates are within the vertical coordinate range according to the coordinate values of the edge point cloud in the camera image coordinate system;

[0082] S34: Project and convert the coordinate values of the filtered edge point cloud in the camera image coordinate system to the camera pixel coordinate system according to the projection conversion relationship between the camera image coordinate system and the camera pixel coordinate system to obtain the edge projection point cloud.

[0083] For the sake of easy understanding, assume that the coordinate value of the edge point cloud in the lidar coordinate system is , and the coordinate value of the edge point cloud in the camera coordinate system is as an example. Thus, each edge point satisfies:

[0084] .

[0085] Furthermore, perform a further normalized conversion on the coordinate value of each edge point of the edge point cloud in the camera coordinate system, that is, convert each edge point to ; Multiply the three-dimensional coordinate value after the normalized plane conversion of the edge point cloud by the internal parameter of the camera , and the edge projection point cloud of the edge point cloud in the camera image coordinate system can be obtained; where is the pixel coordinate value of the origin O1 of the camera coordinate system in the camera pixel coordinate system, [[ID=^]] , 、 are the size of each pixel point on the imaging plane in the camera in the x-axis and y-axis directions respectively, and is the lens focal length of the camera.

[0086] For the sake of easy understanding, the coordinate value of any sampling point in the lidar coordinate system converted to the coordinate value in the camera pixel coordinate system The process can be expressed as:

[0087] ;

[0088] ;

[0089] .

[0090] On this basis, further considering that all pixel points in the scene image collected by the camera must be within the photosensitive range of the camera's imaging plane. Therefore, the point cloud in the area outside the photosensitive range of this imaging plane has no reference value for the edge projection point cloud. Therefore, the horizontal coordinate range and vertical coordinate range of the imaging plane's photosensitive range in the camera image coordinate system can be further determined according to the length and width range interval of the imaging plane's photosensitive range, and the edge projection points not within the area enclosed by this horizontal coordinate range and vertical coordinate range can be removed.

[0091] After removing the edge projection points outside the photosensitive range of the imaging plane, the coordinate values of the remaining edge projection points in the camera image coordinate system can be further discretely converted into the camera pixel coordinate system; the so-called discrete conversion is to, according to the coordinate value of each edge projection point in the camera image coordinate system, fall into each pixel point in the camera pixel coordinate system, and use the coordinate value of the pixel point where each edge projection point is located in the camera pixel coordinate system as the pixel coordinate value of this edge projection point in the camera pixel coordinate system.

[0092] S4: Construct an objective function according to the coordinate distance deviation between each edge projection point and each edge pixel point in the edge projection point cloud in the camera pixel coordinate system.

[0093] Theoretically speaking, when the initial conversion relationship can accurately represent the conversion relationship between the lidar coordinate system and the camera coordinate system, the edge projection point and the edge pixel point determined by the same edge contour point on the target object will coincide in the camera pixel coordinate system; even if the initial conversion relationship cannot accurately represent the conversion relationship between the lidar coordinate system and the camera coordinate system, the edge projection point and the edge pixel point determined by the same edge contour point on the target object should also be adjacent to each other in the camera pixel coordinate system.

[0094] Therefore, the process of constructing the objective function in this embodiment may include:

[0095] S41: According to the coordinate values of each edge projection point in the edge projection point cloud in the camera pixel coordinate system, search for neighboring points of each edge projection point among the edge pixel points to obtain neighboring pixel points corresponding to each edge projection point one by one;

[0096] S42: Perform Euclidean distance operations between each edge projection point and its corresponding neighboring pixel point, and use the sum of the Euclidean distances between all edge projection points and their corresponding neighboring pixel points as the objective function.

[0097] Furthermore, the process of searching for neighboring points for each edge projection point may include:

[0098] Construct a KD-Tree for each edge pixel point, use each edge projection point as a search point, and find the edge pixel point closest to the search point as the neighboring pixel point corresponding to the search point.

[0099] In practical applications, each edge pixel point can be used as a search point to search for the edge pixel point closest to it, which is its neighboring pixel point. In this way, the neighboring pixel points corresponding to each edge projection point one by one can be obtained.

[0100] Therefore, based on the deviation between each edge projection point and its corresponding neighboring pixel point, it is ; where is the total number of edge projection points, is the pixel coordinate value of the th edge projection point, is the pixel coordinate value of the neighboring pixel point corresponding to the th edge projection point; thus, the objective function can be defined as:

[0101] S5: Optimize and iterate each calibration parameter in the initial transformation relationship to obtain the calibration parameter that minimizes the objective function, and use the initial transformation relationship determined by the calibration parameter that minimizes the objective function as the calibration transformation relationship between the lidar and the camera.

[0102] In the actual optimization iteration process, the LEVENBERG-MARQUARDT algorithm can be used as the optimization algorithm to optimize and iterate the objective function; when the norm of the change amount of the calibration parameter in two consecutive optimization iterations is lower than the preset threshold, the iteration process ends; in addition, the preset number of times can be set to 40 times, and the iteration process can also be stopped if the number of iterations reaches the preset number of times.

[0103] To sum up, in this application, based on the data collection of the same scene by using laser radar and camera, the edge point cloud in the scene point cloud collected by the laser radar is screened out, and the edge pixel points in the scene image collected by the camera are identified, which is a more identifying data information; obviously, the edge point cloud and edge pixel points should represent the data information of the edge contour of the same object; theoretically, if the accurate relative posture relationship between the laser radar and the camera can be obtained, that is, the relative transformation relationship between the coordinate systems of the two, after the edge point cloud and the edge pixel points are converted to the same coordinate system, the edge point cloud should be able to coincide with the edge pixel points; based on this, in this application, the relative deviation of the edge point cloud and the edge pixel points converted to the same coordinate system is used as the objective function, and the calibration parameters that characterize the transformation relationship between the laser radar and the camera are repeatedly iterated, and finally the calibration parameters that can minimize the objective function are determined, and the calibration transformation relationship between the laser radar and the camera is determined thereby. During the calibration process of the conversion relationship between the lidar and the camera, this application only needs to use the data information of the contour edges of objects in the scene. This not only reduces the amount of data for the calibration operation to a certain extent, but also does not increase the computational complexity of this application even in the face of extremely complex scenes. Therefore, the calibration difficulty is reduced while ensuring the accuracy of the calibration.

[0104] Based on the above discussion, in an optional embodiment of the present application, the process of identifying edge point clouds in the scene point cloud in S2 may include:

[0105] S21: Obtain the echo sequence number and reflection intensity corresponding to each sampling point according to the echo label corresponding to each sampling point in the scene point cloud;

[0106] S22: Identify, according to the echo sequence number, an initial edge point cloud determined by the first echo in the double echo in the scene point cloud;

[0107] S23: based on the reflection intensity, removing the initial edge points in the initial edge point cloud whose reflection intensity is greater than a set intensity threshold;

[0108] S24: Filter the initial edge point cloud after elimination to obtain an edge point cloud.

[0109] It should be noted that when the lidar collects point clouds of a target object in the form of multiple echoes, the echo information reflected by the same laser beam irradiating the target object is recorded through the echo tag; the echo number and reflection intensity of each sampling point can be obtained through the echo tag.

[0110] If in the multi - echo mode, there are four possible echo sequence numbers: 00, 01, 10, and 11, indicating that the point cloud information at this point is calculated from the 0th, 1st, 2nd, and 3rd echo information respectively; among them, the 0th echo information refers to that the optical system inside the lidar will also generate an echo, which is recorded as the 0th echo. For the reflection intensity of the 3rd echo and subsequent echoes, it is generally relatively weak and can be ignored. Thus, in this embodiment, the scene point cloud is mainly obtained based on the double - echo data obtained from the first echo and the second echo collected by the lidar; of course, it can be understood that when the beam of the lidar irradiates a relatively flat area on the target object, the first echo and the second echo can be fused into the same echo, that is, the collected echo information is single - echo information. Obviously, the point cloud determined by this kind of echo does not belong to the edge point cloud to be screened in this application.

[0111] In the process of screening the edge point cloud, it is possible to determine whether the sampling point is obtained based on the first echo by determining the echo sequence number corresponding to each sampling point in the scene point cloud, and to determine whether there is a second echo obtained by the same laser beam for this sampling point. If there is a corresponding second echo, then this sampling point is the initial edge point determined by the first echo in the double - echo.

[0112] In addition, as Figure 4 shown, when the laser beam output by the lidar is cut by the contour edge of the foreground object, so that a part of the light spot is projected onto the edge area of the foreground object, and the other part is projected onto the background object, the coordinate information of the sampling point determined based on the first echo is the center of the light spot area projected onto the edge of the foreground object; similarly, the position of the sampling point determined by the second echo is the center of the light spot area projected onto the edge of the background object. Then, as Figure 4 shown in the scene of the lidar on the left, the proportion of the laser beam projected onto the foreground object is relatively large, making the light spot area in the edge area of the foreground object relatively large, and the determined sampling point does not exactly fit the contour edge of the foreground object; conversely, as Figure 4 shown in the scene of the lidar on the right, the proportion of the laser beam projected onto the foreground object is relatively small, making the light spot area in the edge area of the foreground object relatively small, and the determined sampling point is closer to the contour edge of the foreground object, and can better represent the edge position point of the foreground object. Thus, it can be seen that there is a certain error in the accuracy of the initial edge point cloud determined only based on the first echo.

[0113] Therefore, in this embodiment, the initial edge point cloud determined based on the first echo is further screened based on the reflection intensity.

[0114] As described above, the coordinate information of the initial edge points is determined based on the size of the light spot area where the laser beam is projected on the contour edge of the foreground object. The larger the light spot area, the greater the error of the determined initial edge points. And the first echo reflected by this light spot area has a relatively large reflected energy, so the reflection intensity is also often large. Therefore, in this embodiment, further screening can be performed based on whether the reflection intensity corresponding to each initial edge point is greater than the set intensity threshold, and only the initial edge point cloud with a reflection intensity less than the set intensity threshold is retained. The set intensity threshold can be set to the average value of the reflection intensities corresponding to all the initial edge point clouds, or slightly greater than the average value of the reflection intensities. There is no specific limitation in this application.

[0115] After obtaining the initial edge point cloud by removing some unreliable initial edge points based on the reflection intensity, further screening can be performed to finally obtain a more accurate edge point cloud.

[0116] In a specific implementation manner of this embodiment, the above S24 can further include:

[0117] S241: Perform a neighboring point search for each initial edge point in the initial edge point cloud after removal to obtain m neighboring points corresponding to each initial edge point;

[0118] S242: Perform a centroid point operation on the m neighboring points corresponding to each initial edge point to obtain the neighboring centroid corresponding to each initial edge point;

[0119] S243: Perform an Euclidean distance operation on each initial edge point and the corresponding neighboring centroid to obtain the neighboring Euclidean distance corresponding to each initial edge point;

[0120] S244: Screen out the initial edge points in the initial edge point cloud whose corresponding neighboring Euclidean distances are not less than the set distance threshold as the edge points in the obtained edge point cloud.

[0121] In practical applications, a KD tree can be constructed for the N initial edge points for efficient nearest neighbor search. Take each initial edge point as the search point in turn, perform a K-nearest neighbor search on other initial edge points, and find the nearest initial edge points of this search point; calculate the centroid of the nearest initial edge points, and the coordinate value of this centroid can be:

[0122] .

[0123] Perform an Euclidean distance operation between the search point and the centroid of the m nearest initial edge points corresponding to it to obtain the adjacent Euclidean distance corresponding to the search point. If the distance between the adjacent Euclidean distance and the search point is relatively large, it indicates that the search point is also in a relatively marginal position among the initial edge points adjacent to it. Therefore, it is more likely to be a position point on the contour edge of the target object. Thus, this search point can be retained as an edge point; conversely, when the distance between the adjacent Euclidean distance and the search point is relatively large, it indicates that the search point is in a relatively central position among the initial edge points adjacent to it, and thus it cannot be a position point on the contour edge of the target object, so this search point can be excluded. Based on the above principle, more accurate and reliable edge point clouds can be further selected from the initial edge point clouds finally.

[0124] In this embodiment, the screening of the initial edge point clouds can not only exclude unreliable initial edge points, but also greatly reduce the computational workload of subsequent data processing.

[0125] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the elements inherent in a process, method, article or device including a series of elements. Without more limitations, the element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element. In addition, the parts of the above technical solutions provided in the embodiments of the present application that are consistent with the corresponding technical solutions in the prior art in terms of implementation principles are not described in detail to avoid excessive elaboration.

[0126] In this article, specific examples are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention. It should be pointed out that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A method for calibrating the parameters of a lidar and a camera, characterized in that, Including: Obtain the scene point cloud and scene image collected by the lidar and camera respectively for the same scene; Identify the edge point cloud in the scene point cloud; And identify the edge pixel points in the scene image; According to the pre-determined initial transformation relationship and the camera internal parameters, convert the coordinate values of the edge point cloud in the lidar coordinate system to the coordinate values in the camera pixel coordinate system to obtain the edge projected point cloud; Construct an objective function according to the coordinate distance deviation between each edge projected point in the edge projected point cloud and each of the edge pixel points in the camera pixel coordinate system; Optimize and iterate each calibration parameter in the initial transformation relationship to obtain the calibration parameter that minimizes the objective function, and use the initial transformation relationship determined by the calibration parameter that minimizes the objective function as the calibration transformation relationship between the lidar and the camera.

2. The method for calibrating the parameters of the lidar and the camera according to claim 1, wherein, Identifying the edge point cloud in the scene point cloud includes: According to the echo label corresponding to each sampling point in the scene point cloud, obtain the echo sequence number and reflection intensity corresponding to each sampling point; According to the echo sequence number, identify the initial edge point cloud determined by the first echo in the double echo in the scene point cloud; According to the reflection intensity, remove the initial edge points in the initial edge point cloud whose reflection intensity is greater than the set intensity threshold; Screen the initial edge point cloud after removal to obtain the edge point cloud.

3. The method for calibrating the parameters of the lidar and the camera according to claim 2, wherein, Screening the initial edge point cloud after removal to obtain the edge point cloud includes: Perform a neighboring point search for each initial edge point in the initial edge point cloud after removal to obtain m neighboring points corresponding to each initial edge point; Perform a centroid point operation on the m neighboring points corresponding to each initial edge point to obtain the neighboring centroid corresponding to each initial edge point; Perform an Euclidean distance operation on each initial edge point and the corresponding neighboring centroid to obtain the neighboring Euclidean distance corresponding to each initial edge point; Screen out the initial edge points in the initial edge point cloud whose corresponding neighboring Euclidean distance is not less than the set distance threshold as the edge points in the obtained edge point cloud.

4. The method for calibrating the parameters of the lidar and the camera according to claim 1, characterized in that, Identifying the edge pixel points in the scene image includes: Perform grayscale processing on the scene image to obtain a grayscale image; Use the Canny operator to extract the edge pixel points in the grayscale image.

5. The method for calibrating the parameters of the lidar and the camera according to claim 1, characterized in that The process of pre-determining the initial transformation relationship includes: According to the relative position relationship between the lidar and the camera, determine the set rotation matrix and the initial translation vector representing the transformation relationship between the lidar coordinate system and the camera coordinate system; Convert the set rotation matrix into an initial rotation matrix representing each matrix element with quaternion parameters; Use the initial rotation matrix and the initial translation vector together as the initial transformation relationship; Correspondingly, according to the pre-determined initial transformation relationship and the camera internal parameters, convert the coordinate values of the edge point cloud in the lidar coordinate system to the coordinate values in the camera pixel coordinate system to obtain the edge projected point cloud, including: Convert the coordinate values of the edge point cloud in the lidar coordinate system to the camera coordinate system according to the initial conversion relationship, and obtain the coordinate values of each edge point of the edge point cloud in the camera coordinate system; According to the camera internal parameters, convert the coordinate values of the edge point cloud in the camera coordinate system to the camera pixel coordinate system to obtain the edge projected point cloud.

6. The method for calibrating parameters of the lidar and the camera according to claim 5, characterized in that, According to the camera internal parameters, converting the coordinate values of the edge point cloud in the camera coordinate system to the camera pixel coordinate system to obtain the edge projected point cloud includes: Perform a normalized plane conversion on the coordinate values of the edge point cloud in the camera coordinate system, and perform a perspective conversion on the edge point cloud according to the normalized converted coordinate values of the edge point cloud and the camera internal parameters to obtain the edge projected point cloud of the edge point cloud in the camera image coordinate system; According to the horizontal coordinate range and vertical coordinate range of the imaging plane photosensitive range of the camera in the camera image coordinate system, filter out the edge projected points in the edge point cloud whose horizontal coordinates are within the horizontal coordinate range and vertical coordinates are within the vertical coordinate range; Perform a discretization conversion on the coordinate values of the filtered edge point cloud in the camera image coordinate system to the camera pixel coordinate system to obtain the pixel coordinate values of the edge projected point cloud in the pixel coordinate system.

7. The method for calibrating the parameters of the lidar and the camera according to claim 5, characterized in that, Optimize and iterate the calibration parameters in the initial conversion relationship, including: Optimize and iterate each quaternion parameter in the initial conversion matrix and each translation parameter in the initial translation vector.

8. The method for calibrating the parameters of the lidar and the camera according to any one of claims 1 to 7, characterized in that, Construct an objective function according to the coordinate distance deviation between each edge projected point in the edge projected point cloud and each edge pixel point in the camera pixel coordinate system, including: According to the coordinate values of each edge projected point in the edge projected point cloud in the camera pixel coordinate system, perform a neighboring point search for each edge projected point among each edge pixel point to obtain the neighboring pixel points corresponding to each edge projected point one by one; Perform an Euclidean distance operation between each edge projected point and the corresponding neighboring pixel point, and use the sum of the Euclidean distances between all the edge projected points and the corresponding neighboring pixel points as the objective function.

9. The method for calibrating the parameters of the lidar and the camera according to claim 8, characterized in that, Performing a neighboring point search for each edge projected point among each edge pixel point to obtain the neighboring pixel points corresponding to each edge point one by one includes: Construct a KD-Tree for each edge pixel point, and use each edge projected point as a search point to find the edge pixel point closest to the search point as the neighboring pixel point corresponding to the search point.

10. The method for calibrating parameters of the lidar and the camera according to claim 8, wherein, Optimizing and iterating the calibration parameters in the initial conversion relationship includes: Optimize and iterate each of the calibration parameters. When the norm of the change amount of the calibration parameters in two consecutive optimization iterations is lower than a preset threshold, or the number of iterations reaches a preset number, the iteration process ends.

Images