Joint calibration method, device, medium and equipment for non-mechanical radar and camera

Abstract

One or more embodiments of the present specification provide a method, device, medium and equipment for joint calibration of a non-mechanical radar and a camera, wherein the method includes: determining a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera; spatially aligning the plurality of first boundary corner points and the plurality of second boundary corner points to convert them to the same coordinate system; matching the plurality of first boundary corner points and the plurality of second boundary corner points in the same coordinate system by a preset algorithm to obtain an optimal rotation matrix and a translation matrix between the plurality of first boundary corner points and the plurality of second boundary corner points. The present invention makes the result of the joint calibration more accurate by matching the boundary corner points.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of information fusion technology, and in particular, to a method, device, medium and equipment for joint calibration of a non-mechanical radar and a camera. Background Art

[0002] Three-dimensional mechanical radar and camera can achieve good visual effects in terms of performance. However, the coordinate systems obtained by these two sensors are different, not the same coordinate system. Data fusion requires that the data of both be placed in the same coordinate system. Therefore, the two need to be jointly calibrated to determine their conversion relationship.

[0003] However, the current joint calibration method generally calibrates the camera first, and then calibrates the radar, and uses the distance correspondence principle combined with the plane normal vector of the fitting plane to solve the joint calibration result. However, this method has the disadvantage that the two distances need to be measured separately, and the final joint calibration result is quite different from the actual one, and the accuracy is insufficient. At present, there is no method or device that can solve the above problem. Summary of the invention

[0004] In view of this, the purpose of one or more embodiments of the present specification is to propose a joint calibration method, device, medium and equipment for a non-mechanical radar and a camera to solve the problem that the results of the joint calibration are significantly different from the actual results.

[0005] Based on the above purpose, one or more embodiments of this specification provide a joint calibration method of a non-mechanical radar and a camera, including:

[0006] Determine a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera;

[0007] spatially aligning the plurality of first boundary corner points and the plurality of second boundary corner points to convert them into the same coordinate system;

[0008] The plurality of first boundary corner points and the plurality of second boundary corner points in the same coordinate system are matched by a preset algorithm to obtain an optimal rotation matrix and a translation matrix between the plurality of first boundary corner points and the plurality of second boundary corner points.

[0009] In combination with the above description, in another possible implementation of the embodiment of the present invention, spatially aligning the plurality of first boundary corner points and the plurality of second boundary corner points to convert them into the same coordinate system includes:

[0010] Determining the coordinate correspondence between the coordinate systems of the non-mechanical radar and the camera according to the intrinsic parameter calibration of the camera;

[0011] The first boundary corner points and the second boundary corner points are spatially aligned with each other through the coordinate correspondence in combination with a time alignment operation, so as to be converted into the same coordinate system.

[0012] In combination with the above description, in another possible implementation manner of the embodiment of the present invention, the determining of a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera includes:

[0013] Determining two groups of four corner points of the non-mechanical radar as the plurality of first boundary corner points;

[0014] Two groups of four corner points of the camera are determined as the plurality of second boundary corner points.

[0015] In combination with the above description, in another possible implementation manner of the embodiment of the present invention, the determining of a plurality of first boundary corner points of the non-mechanical radar includes:

[0016] When preprocessing the point cloud of the non-mechanical radar, determining whether the corner point data is in the area of ​​interest of the non-mechanical radar;

[0017] When the judgment result is yes, retaining the corner point data, and further determining the plurality of first boundary corner points from the corner point data;

[0018] When the judgment result is no, the corner point data is deleted.

[0019] In combination with the above description, in another possible implementation of the embodiment of the present invention, the method further includes:

[0020] When the point cloud data is preprocessed, it includes:

[0021] Performing rasterization processing on the point cloud data, and converting the three-dimensional point cloud data into two-dimensional point cloud data;

[0022] Set the length L and width W of the grid, distribute the two-dimensional point cloud data to each grid, and generate a two-dimensional grid map G. The initial state of the grid is S(G (x,y) )=default;

[0023] For each grid in G, when the grid is empty, the grid state is set to S(G (x,y) )=free; when the grid is not empty, the depth difference of the grid is calculated, which is the mean of the interpolation sum of the two points with the largest value and the two points with the smallest value on the x-coordinate of the two-dimensional coordinate system, and is magnified by α times, that is, Δd(G (x,y) )=α(x max -x min +x least_max -x least_min ) / 2;

[0024] Calculate the average depth of the grid, which is the x-coordinate of all the point clouds in the grid. At the same time, we also need to count the number of continuous empty grids on the left and right of the grid E left (G (x,y) ), E right (G (x,y) );

[0025] Traverse the grid row by row and set the edge grid that meets the following three conditions to state S(G (x,y) )=border:

[0026] Average depth of adjacent grids If there is a sudden change, it is determined to be an edge grid: or

[0027] There is a continuous ξ on the left or right side of the grid n If there are more than 100 empty grids, it is considered as an edge grid: max(E left (G (x,y) ),E right (G (x,y) ))>ξ n ;

[0028] The depth difference of the grid is greater than the threshold ξ Δ , then it is determined to be an edge grid: Δd(G (x,y) )≥ξ Δ This condition takes into account the existence of boundaries within the grid, that is, the depth difference of the point cloud within the grid is greater than a certain threshold;

[0029] According to the grid status bit S(G (x,y) ) will S(G (x,y) )=The three-dimensional points of the point cloud in the border grid are stored in the newly created point cloud set C potential , as a potential boundary point cloud, the potential boundary point cloud is used to extract the plurality of first boundary corner points.

[0030] In combination with the above description, in another possible implementation of the embodiment of the present invention, the method further includes:

[0031] Performing multiple plane fitting on the point cloud set according to the number of calibration plates used, filtering out interference noise points contained therein, so that the plurality of first boundary corner points are on the same plane;

[0032] The spacing between the four corner points of a single calibration plate is calculated. When the spacing is greater than a preset spacing threshold, the corner point data of the calibration plate is discarded. When the spacing is less than the preset spacing threshold, the corner point data corresponding to the calibration plate is saved to act on the first boundary corner points.

[0033] In a second aspect, an exemplary embodiment of the present invention further provides a non-mechanical radar and camera joint calibration device, including:

[0034] A determination module, used to determine a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera;

[0035] A conversion module, used for spatially aligning the plurality of first boundary corner points and the plurality of second boundary corner points so as to convert them into the same coordinate system;

[0036] The matching module is used to match the first boundary corner points and the second boundary corner points in the same coordinate system through a preset algorithm to obtain the optimal rotation matrix and translation matrix between the first boundary corner points and the second boundary corner points.

[0037] The above-mentioned device, the conversion module, includes:

[0038] A calibration submodule, used to determine the coordinate correspondence between the coordinate systems of the non-mechanical radar and the camera according to the intrinsic calibration of the camera;

[0039] The alignment submodule is used to spatially align the first boundary corner points and the second boundary corner points through the coordinate correspondence in combination with a time alignment operation, so as to convert them into the same coordinate system.

[0040] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the joint calibration method of the non-mechanical radar and the camera when executing the program.

[0041] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the non-mechanical radar and camera joint calibration method.

[0042] From the above, it can be seen that the joint calibration method, device, medium and equipment of the non-mechanical radar and camera provided by one or more embodiments of the present specification, through iterative matching of multiple groups of first boundary corner points and second boundary corner points, can ensure that the selected boundary corner points have high reliability, and ultimately minimize the error and reduction of the positions of all corner points of the laser radar corner points and the camera. When the optimal rotation matrix and translation matrix obtained based on each coordinate are converted to the laser radar coordinate system, the errors in the three directions of xyz are smaller. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate one or more embodiments of the present specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only one or more embodiments of the present specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of a joint calibration example of one or more embodiments of this specification;

[0045] Figure 2 A basic flow chart of a joint calibration method of a non-mechanical radar and a camera according to one or more embodiments of this specification;

[0046] Figure 3 A schematic diagram of a coordinate system conversion process for one or more embodiments of this specification;

[0047] Figure 4 A schematic diagram of a point cloud processing flow chart of one or more embodiments of this specification;

[0048] Figure 5 An example schematic diagram of a calibration plate for one or more embodiments of this specification;

[0049] Figure 6 A schematic diagram of a boundary point cloud processing flow of one or more embodiments of this specification;

[0050] Figure 7 A schematic diagram of a corner point determination process of one or more embodiments of this specification;

[0051] Figure 8 A schematic diagram of an iterative calculation process for one or more embodiments of this specification;

[0052] Fig. 9 A schematic diagram of the structure of a joint calibration device for a non-mechanical radar and a camera according to one or more embodiments of this specification;

[0053] Fig.10 A schematic diagram of the device structure of one or more embodiments of this specification. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0055] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of this specification should be understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in one or more embodiments of this specification do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprises" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0056] Combination Figure 1 As shown, the non-mechanical radar (hereinafter referred to as radar, lidar) and the camera are calibrated on the same calibration plate. The point cloud data of the radar is in one coordinate system, and the visual data of the camera is in another coordinate system. The two coordinate systems need to be rigidly transformed so that the data obtained can be in the same coordinate system, so that when the converted coordinate system is used, the error with the original coordinate system can be minimized or basically eliminated.

[0057] One or more embodiments of this specification, combined with Figure 2 The basic schematic diagram of the method flow of the present invention shown includes the following steps:

[0058] In step 210, a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera are determined;

[0059] The non-mechanical radar is generally a laser radar, which can form point cloud data, and the camera can form visual pose data.

[0060] In step 220, the plurality of first boundary corner points and the plurality of second boundary corner points are spatially aligned to be converted into the same coordinate system;

[0061] In step 230, the plurality of first boundary corner points and the plurality of second boundary corner points in the same coordinate system are matched by a preset algorithm to obtain an optimal rotation matrix and a translation matrix between the plurality of first boundary corner points and the plurality of second boundary corner points.

[0062] Different from the existing method of transforming the spatial coordinate system by fitting the plane normal vector, the method of the present invention matches by determining the specific boundary corner points, and then minimizes the error of the optimal rotation matrix and translation matrix obtained at the boundary corner points in the same coordinate system in the three-dimensional coordinate system, further improving the accuracy of the converted coordinates.

[0063] As a feasible implementation of an exemplary embodiment of the present invention, as shown below, the subscript c represents the camera coordinate system (camera, camera), and the subscript l represents the laser radar coordinate system (lidar, radar). Then the camera coordinates of a point in space are (x c ,y c , z c ), the camera coordinates on the two-dimensional plane are (x c ,y c ), similarly, the radar coordinates of a point in space are (x l ,y l , z l ), the radar coordinates of the two-dimensional plane are (x l ,y l ).

[0064] The transformation relationship between the two coordinate systems can be expressed as follows:

[0065]

[0066] Among them, R represents a 3*3 matrix, r represents a parameter in the R matrix, T represents a translation vector, and t is a parameter in the translation vector.

[0067] In the implementation scenario of the exemplary embodiment of the present invention, combined with Figure 3 As shown, after the parameter configuration, the point cloud data of the radar is parsed by its exclusive radar data parsing module, and the image frame data obtained by the camera is parsed by its exclusive camera video stream acquisition module. After the time synchronization (alignment) operation of the data is performed based on the timestamp information carried by each data, the posture transformation relationship is solved according to the preset algorithm to obtain the rigid body transformation relationship of its coordinate system.

[0068] The methods applied in the above scenarios include lidar raw data analysis, camera video stream acquisition, data time alignment, visual pose recognition, lidar pose extraction and pose conversion relationship solution.

[0069] In the implementation scenario of the exemplary embodiment of the present invention, the visual pose recognition of the camera includes reading parameter initialization from the configuration file, identifying the markers on the calibration plate, calculating the corner points of the markers according to the internal EPnP (Perspective nPoint, 3D to 2D conversion) algorithm, and calculating the corner points of the calibration plate boundary. The laser radar pose extraction includes laser radar point cloud preprocessing, point cloud boundary extraction, point cloud boundary straight line fitting, and calculating the corner points of the calibration plate boundary.

[0070] Combination Figure 4As shown, after acquiring the point cloud data of the radar, preprocessing is performed, the point cloud boundary is extracted, and the calibration plate corner points are calculated after linear fitting of the point cloud boundary, thereby determining several first boundary corner points required by the method of the present invention, which are generally two groups of 4 corner points with a total of 8 corner points.

[0071] In the implementation scenario of the exemplary embodiment of the present invention, a laser radar analysis module can be used to analyze point cloud data. It is generally responsible for analyzing the point cloud data of the corresponding model of laser radar. The module can receive and analyze point cloud data of non-mechanical radars and is compatible with mechanical radars. The analysis module can basically cover common high-performance three-dimensional laser radars, with a wide range of applications and good hardware coverage.

[0072] In the implementation scenario of the exemplary embodiment of the present invention, a camera video stream acquisition module is used to parse visual image data. It is generally responsible for capturing video stream data from the camera, acquiring the stream according to the camera type, and converting the camera's original video stream into multiple frames of pictures. The picture data also carries a timestamp to facilitate subsequent time alignment based on the established time alignment module.

[0073] Since the sampling initial time and sampling frequency are different between different radars and cameras, the time alignment module can be used to align the two data times. Since the calibration object and the sensor are in a relatively static position during calibration, the method of the present invention can adopt a simpler strategy for aligning the sensor data with a high sampling rate. A new container will be created for sensor data with a high sampling rate, and for other sensor data, a container with the closest sampling time will be found for storage. Only the data that has been successfully stored in correspondence with the two sensor data will be used for subsequent algorithm calculations. The existence of the time alignment module eliminates the occurrence of scenarios with large errors in the two sensor scenes with a lower computing burden. The maximum sampling time difference between the two sensor data used for calibration is 20ms.

[0074] The visual pose recognition module of the camera determines the pose of the marker by identifying the marker on the calibration plate, and then infers the pose of the corner point of the calibration plate by referring to the design of the calibration plate. The pose recognition of the marker is to extract the two-dimensional coordinates (x 2D ,y 2D ), combined with the known camera internal parameters, the three-dimensional coordinates of the corner points are converted into PnP problems (Perspective N points). When the number of points in the known two-dimensional coordinate system is >3, the problem can be solved. Here we use the EPnP algorithm to solve the problem. Compared with other methods, the complexity of the EPnP method is O(n), which is very efficient for PnP problems with a large number of point pairs. At the same time, this module can adapt to a variety of markers and has strong adaptability. Marker design such as Figure 5 shown.

[0075] Combination Figure 6 In the boundary extraction process shown in the implementation scenario of the exemplary embodiment of the present invention, the laser radar point cloud preprocessing sets the region of interest (ROI) to determine the area that the algorithm needs to process. After extracting the calibration plate data perpendicular to the distance direction according to the ROI, the point cloud data is stored in the corresponding grid, the grid attributes are calculated, and the grid is traversed row by row to determine whether it meets the conditions. After determining whether it belongs to the candidate boundary, the distance between all points in the single point cloud set and the fitting plane in the fitting plane is greater than the preset threshold. After noise removal, the final boundary set is obtained. This process will be described in detail below:

[0076] Set the minimum coordinate x of the area to be detected in the x, y, and z directions min ,y min 、z min and the maximum value x max ,y max 、z max For a laser point with coordinates (x, y, z), if x min ≤x≤x max ,y min ≤y≤y max 、z min ≤z≤z max , then the point is within the region of interest and is retained; otherwise, the point is considered to be outside the region of interest and is deleted.

[0077] In an implementation of an exemplary embodiment of the present invention, for the corner points of radar data and camera data, two groups of four corner points, totaling eight corner points, are extracted as their boundary corner points, which can achieve a good accuracy improvement effect.

[0078] The point cloud boundary extraction of LiDAR is suitable for non-mechanical scanning radars with wireless beam information, and is also backward compatible with mechanical LiDARs. For the pre-processed point cloud, the point cloud is first rasterized to convert the three-dimensional point cloud into a two-dimensional image. The two-dimensional image does not have depth of field information, that is, it is located on the yz plane. Set the length L and width W of the grid, assign the point cloud to each grid, and generate a two-dimensional grid map G, that is, each grid in G contains several laser points, and the initial state of the grid is S(G (x,y) )=default.

[0079] For each grid in G, if the grid is empty, set the grid state to S(G (x,y))=free, if the grid is not empty, the depth difference of the grid is calculated (the average of the interpolation of the two points with the largest values ​​and the two points with the smallest values ​​on the x-coordinate) and amplified by α, that is, Δd(G (x,y) )=α(x max -x min +x least_max -x least_min ) / 2, we also need to calculate the average depth (x coordinate) of the grid, that is, the average depth of all point clouds in the grid. At the same time, we also need to count the number of continuous empty grids on the left and right of the grid E left (G (x,y) ), E right (G (x,y) ).

[0080] Traverse the grid row by row and set the edge grid that meets the following three conditions to state S(G (x,y) )=border.

[0081] i) Average depth of adjacent grids If there is a sudden change, it is determined to be an edge grid: or This condition takes into account the normal situation that there should be mutations in the grid occupied by the boundary of the calibration plate and the mental information of the surrounding environment.

[0082] ii) There is a continuous ξ on the left or right side of the grid n If there are more than 100 empty grids, it is considered as an edge grid: max(E left (G (x,y) ),E right (G (x,y) ))>ξ n This condition takes into account the possibility of empty grids in some areas due to ROI area extraction (mainly cutting in the x direction). At the same time, due to the density of the point cloud and the size of the grid during the rasterization process, the grid generated in the calibration plate may also have no points. Therefore, it is necessary to satisfy the condition that there are continuous empty grids greater than a certain threshold.

[0083] iii) The depth difference of the grid is greater than the threshold ξ Δ , then it is determined to be an edge grid: Δd(G (x,y) )≥ξ Δ This condition takes into account the existence of boundaries within the grid, that is, the depth difference of the point cloud within the grid is greater than a certain threshold.

[0084] According to the grid status bit S(G (x,y) ) will S(G (x,y) )=The three-dimensional points of the point cloud in the border grid are stored in the newly created point cloud set C potential, as a potential boundary point cloud.

[0085] According to the number of calibration plates used, the point cloud set is fitted with planes N times. The specific operation is to cluster the point cloud, select the N clusters with the largest number of points, perform N RANSAC plane fittings on each cluster, and store the points used to fit the corresponding planes into the corresponding point set C. planeN After fitting, a second judgment will be made to find C based on the plane parameters obtained by plane fitting. planeN The distance from the point in to the fitted plane is greater than the threshold, that is, d(C planeN (i),planeN)>ξ p2p When planeN The points that have not been eliminated are retained. planeN Re-perform plane fitting. Finally, the point set C planeN The points in the point set C are projected onto the fitted plane, that is, the parameters of the fitted plane are Ax+By+Bz+D=0, and the point set C planeN The point in is P i '(x',y',z'),P i '∈C planeN , the projection point in plane N is P i (x,y,z), then the straight line P i P i 'Normal vector to the plane Parallel, straight line P i P i The parameter equation of ' is Substitute the point (x, y, z) into the plane equation and find t: Substituting t into the parametric equation of the line, we can find the projection point P. i (x,y,z), add the projection point result to the new point set middle.

[0086] After all plane operations are completed, clear C potential , and store the points back in it. After the boundary points are extracted, there will still be interfering non-boundary point noise points. Some of the noise points can be well filtered out through plane fitting, point-to-fitting plane distance determination and quadratic fitting. Finally, the projection of the point to the plane will make the position of the subsequent extracted calibration plate corner points more accurate on a plane, thus affecting the final matching result. The entire edge point cloud extraction process is more stable and more robust.

[0087] Combination Figure 7The schematic diagram of straight line fitting of the point cloud boundary is shown. The point cloud is projected onto a 2D plane according to the camera intrinsic parameters, the 2D point cloud is displayed, the point cloud data is manually selected, and a straight line is fitted to the point cloud data. The straight lines intersect each other to obtain the corner points of the calibration plate, and the spacing of the corner points is calculated. The spacing of the four corner points of a single calibration plate is calculated. When the spacing is greater than a preset spacing threshold, the corner point data of the calibration plate is discarded. When the spacing is less than the preset spacing threshold, the corner point data corresponding to the calibration plate is saved.

[0088] In the process of linear fitting of the point cloud boundary by the first boundary corner points, C potential The 3D point cloud is calibrated based on the camera's intrinsic parameters and projected onto a 2D plane of the same size as the camera screen. Use the graphical interface to select the 4×N edges corresponding to the N calibration plates in turn and store the resulting point clouds in In the point cloud, the corresponding straight line is fitted according to the points in the point cloud.

[0089] Compared with the method of directly extracting corner points from point clouds, this method is more stable and can extract corner points more accurately. The method of directly extracting corner points will fail because the laser radar scanning points do not scan the corner points of the calibration plate, while this method can still accurately fit the edges even when some edge points are not scanned, and then intersect to obtain boundary points.

[0090] The adjacent straight lines in the same calibration plate are intersected to obtain corner points, so as to find the corner points of all calibration plates. Here, the restriction condition of the corner point distance is also set. If the corner point distance is far from the true value, the frame data will be abandoned.

[0091] The pose conversion relationship solving module is responsible for calculating the rotation matrix R between the lidar and the camera. 3×3 and the translation matrix T 3×1 After obtaining two sets of corresponding corner points, the ICP (Iterative Closet Point) point matching method can be used to solve [R|T]. The ICP method minimizes the distance difference between the two sets of points in 3D space. To calculate the result, P and Q are the coordinates of the corner points in the laser radar and camera coordinate systems respectively.

[0092] The general ICP point cloud matching algorithm considers the nearest point of the point as the corresponding point. In the context of coordinate matching, this method of aligning two point sets by minimizing the Euclidean distance between the corresponding points of the two point sets may result in inaccurate results, that is, the correspondence between the points in the two point sets is inconsistent with what is imagined. As an optimization solution, we will predetermine the correspondence between the points in the two sets of point sets before matching the point sets, and then match the points to ensure that the results obtained are the optimal solution under our assumptions and are what we need. According to the Kabsch algorithm, the optimal rotation matrix R of the two point sets to be aligned is calculated, and this method is used to solve the optimal rotation matrix of the two point sets to minimize the root mean square deviation. The translation matrix T can be calculated based on the obtained rotation matrix R.

[0093] Combination Figure 8 As shown, a more specific schematic diagram of the basic process of joint calibration of an exemplary embodiment of the present invention is shown. In order to minimize the result error of the posture conversion relationship, multiple sets of data samples are taken for the calculation of the rotation matrix and the translation matrix, so as to obtain the most accurate and less noisy rigid body transformation results. Multiple sets of data samples are implemented through N iterations on the basis of maintaining the positions of the laser radar and the camera. Each set of samples will be used to calculate [R|T], and then we will average them. Since the rotation matrix cannot be directly averaged, the rotation matrix will first be converted to a quaternion and then averaged. After processing, it will be converted back to the form of a rotation matrix for normalization.

[0094] The beneficial effects of the present invention include:

[0095] The point cloud of non-mechanical radar is rasterized, and the rasterized horizontal grid (row) is regarded as a beam, so as to perform point cloud processing. This solves a series of algorithmic problems caused by the lack of beam information in non-mechanical laser radar. In addition, the calculation of specific grid state parameters in the grid makes it possible to select multiple conditions.

[0096] Through multiple conditions such as sudden changes in the depth of adjacent grids, adjacent empty grids exceeding a certain threshold, and the depth difference of the same grid exceeding the threshold, the potential edge grids are judged and screened, basically covering all scenarios where the grid may be the boundary.

[0097] By clustering the boundary points, plane fitting, and point-to-plane distance calculation and screening, some noise points that are not on the edge of the calibration plate are removed.

[0098] By refitting the plane after removing noise points and projecting the points onto the plane, the accuracy and reliability of the subsequent calibration plate edge straight line fitting and corner point calculation are improved.

[0099] The algorithm is more stable by extracting the boundary first and then extracting the corner points from the boundary.

[0100] The corner point distance re-verification strategy will discard scenes where the corner point distance is too different from the actual size of the calibration plate.

[0101] A simple and reliable time alignment strategy.

[0102] The semi-automatic solution saves some manpower while retaining the accuracy of manual selection. The algorithm is used to select the boundary point cloud as a candidate, so that humans do not have to select boundary points from a large number of difficult-to-distinguish points. At the same time, the last step of manually selecting boundary points from the candidates is a good check, and the selected boundary points have high reliability.

[0103] In an alternative implementation of the exemplary embodiment of the present invention, the ROI area selection does not delete the point cloud, and only the selected point cloud is operated to implement the present invention. Therefore, the method of not using ROI is also within the protection scope of the claims of the present invention.

[0104] In an alternative implementation of an exemplary embodiment of the present invention, RANSAC fitting is not used for plane fitting or line fitting, and other fitting methods can also be used to implement the present invention, so using other fitting methods is an alternative solution.

[0105] It should be noted that the method of one or more embodiments of this specification can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of one or more embodiments of this specification, and the multiple devices will interact with each other to complete the described method.

[0106] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0107] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, one or more embodiments of this specification also provide a joint calibration device for a non-mechanical radar and a camera.

[0108] refer to Fig. 9 The non-mechanical radar and camera joint calibration device comprises:

[0109] A determination module 910 is used to determine a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera;

[0110] A conversion module 920, configured to spatially align the plurality of first boundary corner points and the plurality of second boundary corner points so as to convert them into the same coordinate system;

[0111] The matching module 930 is used to match the first boundary corner points and the second boundary corner points in the same coordinate system through a preset algorithm to obtain the optimal rotation matrix and translation matrix between the first boundary corner points and the second boundary corner points.

[0112] The device of the exemplary embodiment of the present invention, the conversion module 920, includes:

[0113] A calibration submodule, used to determine the coordinate correspondence between the coordinate systems of the non-mechanical radar and the camera according to the intrinsic calibration of the camera;

[0114] The alignment submodule is used to spatially align the first boundary corner points and the second boundary corner points through the coordinate correspondence in combination with a time alignment operation, so as to convert them into the same coordinate system.

[0115] In the apparatus of the exemplary embodiment of the present invention, the determining module is specifically configured to:

[0116] Determining two groups of four corner points of the non-mechanical radar as the plurality of first boundary corner points;

[0117] Two groups of four corner points of the camera are determined as the plurality of second boundary corner points.

[0118] In the device of the exemplary embodiment of the present invention, the determining module is further used for:

[0119] When preprocessing the point cloud of the non-mechanical radar, determining whether the corner point data is in the area of ​​interest of the non-mechanical radar;

[0120] When the judgment result is yes, retaining the corner point data, and further determining the plurality of first boundary corner points from the corner point data;

[0121] When the judgment result is no, the corner point data is deleted.

[0122] The device of the exemplary embodiment of the present invention further includes a pre-processing module, which is used to:

[0123] When the point cloud data is preprocessed, it includes:

[0124] Performing rasterization processing on the point cloud data, and converting the three-dimensional point cloud data into two-dimensional point cloud data;

[0125] Set the length L and width W of the grid, distribute the two-dimensional point cloud data to each grid, and generate a two-dimensional grid map G. The initial state of the grid is S(G (x,y) )=default;

[0126] For each grid in G, when the grid is empty, the grid state is set to S(G (x,y) )=free; when the grid is not empty, the depth difference of the grid is calculated, which is the mean of the interpolation sum of the two points with the largest value and the two points with the smallest value on the x-coordinate of the two-dimensional coordinate system, and is magnified by α times, that is, Δd(G (x,y) )=α(x max -x min +x least_max -x least_min ) / 2;

[0127] Calculate the average depth of the grid, which is the x-coordinate of all the point clouds in the grid. At the same time, we also need to count the number of continuous empty grids on the left and right of the grid E left (G (x,y) ), E right (G (x,y) );

[0128] Traverse the grid row by row and set the edge grid that meets the following three conditions to state S(G (x,y) )=border:

[0129] Average depth of adjacent grids If there is a sudden change, it is determined to be an edge grid: or

[0130] There is a continuous ξ on the left or right side of the grid n If there are more than 100 empty grids, it is considered as an edge grid: max(E left (G (x,y) ),E right (G (x,y) ))>ξ n ;

[0131] The depth difference of the grid is greater than the threshold ξ Δ , then it is determined to be an edge grid: Δd(G (x,y) )≥ξ Δ This condition takes into account the existence of boundaries within the grid, that is, the depth difference of the point cloud within the grid is greater than a certain threshold;

[0132] According to the grid status bit S(G (x,y) ) will S(G (x,y))=The three-dimensional points of the point cloud in the border grid are stored in the newly created point cloud set C potential , as a potential boundary point cloud, the potential boundary point cloud is used to extract the plurality of first boundary corner points.

[0133] The device also includes a judging module, which is used to:

[0134] Performing multiple plane fitting on the point cloud set according to the number of calibration plates used, filtering out interference noise points contained therein, so that the plurality of first boundary corner points are on the same plane;

[0135] The spacing between the four corner points of a single calibration plate is calculated. When the spacing is greater than a preset spacing threshold, the corner point data of the calibration plate is discarded. When the spacing is less than the preset spacing threshold, the corner point data corresponding to the calibration plate is saved to act on the first boundary corner points.

[0136] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more embodiments of this specification, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0137] The non-mechanical radar and camera joint calibration device of the above embodiment is used to implement the corresponding non-mechanical radar and camera joint calibration method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0138] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, one or more embodiments of this specification also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the joint calibration method of the non-mechanical radar and camera described in any of the above embodiments is implemented.

[0139] Fig.10 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0140] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0141] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0142] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0143] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0144] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0145] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0146] The electronic device of the above embodiment is used to implement the corresponding non-mechanical radar and camera joint calibration method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0147] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, one or more embodiments of the present specification also provide a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the joint calibration method of the non-mechanical radar and camera as described in any of the above embodiments.

[0148] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0149] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the joint calibration method of the non-mechanical radar and camera as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0150] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. In line with the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present specification as described above, which are not provided in detail for the sake of simplicity.

[0151] In addition, to simplify the description and discussion, and in order not to obscure one or more embodiments of the present specification, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device may be shown in the form of a block diagram so as to avoid obscuring one or more embodiments of the present specification, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which one or more embodiments of the present specification will be implemented (i.e., these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that one or more embodiments of the present specification may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0152] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0153] One or more embodiments of this specification are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification should be included in the scope of protection of this disclosure.

Claims

1. A joint calibration method for non-mechanical radar and camera, characterized in that: include: Determine a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera; spatially aligning the plurality of first boundary corner points and the plurality of second boundary corner points to convert them into the same coordinate system; Matching the first boundary corner points and the second boundary corner points in the same coordinate system by a preset algorithm to obtain an optimal rotation matrix and a translation matrix between the first boundary corner points and the second boundary corner points; The determining of a plurality of first boundary corner points of the non-mechanical radar comprises: When preprocessing the point cloud of the non-mechanical radar, determining whether the corner point data is in the area of ​​interest of the non-mechanical radar; When the judgment result is yes, retaining the corner point data, and further determining the plurality of first boundary corner points from the corner point data; When the judgment result is no, deleting the corner point data; When the point cloud data is preprocessed, it includes: Performing rasterization processing on the point cloud data, and converting the three-dimensional point cloud data into two-dimensional point cloud data; Set the length L and width W of the grid, distribute the two-dimensional point cloud data to each grid, and generate a two-dimensional grid map G. The initial state of the grid is S(G (x,y) )=default; For each grid in G, when the grid is empty, the grid state is set to S(G (x,y) )=free; when the grid is not empty, the depth difference of the grid is calculated, which is the mean of the interpolation sum of the two points with the largest value and the two points with the smallest value on the x-coordinate of the two-dimensional coordinate system, and is magnified by α times, that is, Δd(G (x,y) )=α(x max -x min +x least_max -x least_min ) / 2; Calculate the average depth of the grid, which is the x-coordinate of all the point clouds in the grid. At the same time, we also need to count the number of continuous empty grids on the left and right of the grid E left (G (x,y) ), E right (G (x,y) ); Traverse the grid row by row and set the edge grid that meets the following three conditions to state S(G (x,y) )=border: Average depth of adjacent grids If there is a sudden change, it is determined to be an edge grid: or There is a continuous ξ on the left or right side of the grid n If there are more than 100 empty grids, it is considered as an edge grid: max(E left (G (x,y) ),E right (G (x,y) ))>ξ n ; The depth difference of the grid is greater than the threshold ξ Δ , then it is determined to be an edge grid: Δd(G (x,y) )≥ξ Δ ; This condition takes into account the existence of boundaries within the grid, that is, the depth difference of the point cloud within the grid is greater than a certain threshold; According to the grid status bit S(G (x,y) )S(G (x,y) )=The three-dimensional points of the point cloud in the border grid are stored in the newly created point cloud set C potential , as a potential boundary point cloud, the potential boundary point cloud is used to extract the plurality of first boundary corner points.

2. The method according to claim 1, characterized in that The spatially aligning the plurality of first boundary corner points and the plurality of second boundary corner points to convert them into the same coordinate system includes: Determining the coordinate correspondence between the coordinate systems of the non-mechanical radar and the camera according to the intrinsic parameter calibration of the camera; The first boundary corner points and the second boundary corner points are spatially aligned with each other through the coordinate correspondence in combination with a time alignment operation, so as to be converted into the same coordinate system.

3. The method according to claim 1, characterized in that The determining of a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera comprises: Determining two groups of four corner points of the non-mechanical radar as the plurality of first boundary corner points; Two groups of four corner points of the camera are determined as the plurality of second boundary corner points.

4. The method according to claim 1, characterized in that: The method further comprises: Performing multiple plane fitting on the point cloud set according to the number of calibration plates used, filtering out interference noise points contained therein, so that the plurality of first boundary corner points are on the same plane; The spacing between the four corner points of a single calibration plate is calculated. When the spacing is greater than a preset spacing threshold, the corner point data of the calibration plate is discarded. When the spacing is less than the preset spacing threshold, the corner point data corresponding to the calibration plate is saved to act on the first boundary corner points.

5. A non-mechanical radar and camera joint calibration device, characterized in that: include: A determination module, used to determine a plurality of first boundary corner points of the non-mechanical radar and a plurality of second boundary corner points of the camera; A conversion module, used for spatially aligning the plurality of first boundary corner points and the plurality of second boundary corner points so as to convert them into the same coordinate system; A matching module, used for matching the plurality of first boundary corner points and the plurality of second boundary corner points in the same coordinate system by a preset algorithm to obtain an optimal rotation matrix and a translation matrix between the plurality of first boundary corner points and the plurality of second boundary corner points; The determining of a plurality of first boundary corner points of the non-mechanical radar comprises: When preprocessing the point cloud of the non-mechanical radar, determining whether the corner point data is in the area of ​​interest of the non-mechanical radar; When the judgment result is yes, retaining the corner point data, and further determining the plurality of first boundary corner points from the corner point data; When the judgment result is no, deleting the corner point data; When the point cloud data is preprocessed, it includes: Performing rasterization processing on the point cloud data, and converting the three-dimensional point cloud data into two-dimensional point cloud data; Set the length L and width W of the grid, distribute the two-dimensional point cloud data to each grid, and generate a two-dimensional grid map G. The initial state of the grid is S(G (x,y) )=default; For each grid in G, when the grid is empty, the grid state is set to S(G (x,y) )=free; when the grid is not empty, the depth difference of the grid is calculated, which is the mean of the interpolation sum of the two points with the largest value and the two points with the smallest value on the x-coordinate of the two-dimensional coordinate system, and is magnified by α times, that is, Δd(G (x,y) )=α(x max -x min +x least_max -x least_min ) / 2; Calculate the average depth of the grid, which is the x-coordinate of all the point clouds in the grid. At the same time, we also need to count the number of continuous empty grids on the left and right of the grid E left (G (x,y) ), E right (G (x,y) ); Traverse the grid row by row and set the edge grid that meets the following three conditions to state S(G (x,y) )=border: Average depth of adjacent grids If there is a sudden change, it is determined to be an edge grid: or There is a continuous ξ on the left or right side of the grid n If there are more than 100 empty grids, it is considered as an edge grid: max(E left (G (x,y) ),E right (G (x,y) ))>ξ n ; The depth difference of the grid is greater than the threshold ξ Δ , then it is determined to be an edge grid: Δd(G (x,y) )≥ξ Δ ; This condition takes into account the existence of boundaries within the grid, that is, the depth difference of the point cloud within the grid is greater than a certain threshold; According to the grid status bit S(G (x,y) )S(G (x,y) )=The three-dimensional points of the point cloud in the border grid are stored in the newly created point cloud set C potential , as a potential boundary point cloud, the potential boundary point cloud is used to extract the plurality of first boundary corner points.

6. The device according to claim 5, characterized in that The conversion module comprises: A calibration submodule, used to determine the coordinate correspondence between the coordinate systems of the non-mechanical radar and the camera according to the intrinsic calibration of the camera; The alignment submodule is used to spatially align the first boundary corner points and the second boundary corner points through the coordinate correspondence in combination with a time alignment operation, so as to convert them into the same coordinate system.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the non-mechanical radar and camera joint calibration method according to any one of claims 1 to 4 is implemented.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the joint calibration method of the non-mechanical radar and camera as described in any one of claims 1 to 4.

Images