A radar and camera calibration and coordinate conversion method

Abstract

This invention discloses a calibration and coordinate transformation method for radar and camera. The calibration method includes: acquiring at least one frame of point cloud data and corresponding image data for each point cloud data; processing the point cloud data for each frame to obtain point cloud data corresponding to a calibration board; translating and rotating the point cloud data corresponding to the calibration board to a preset plane to obtain corresponding first point cloud data; clustering each first point cloud data to obtain at least one cluster; determining the contour point coordinates of each cluster based on the second point cloud data in each cluster; determining a contour point list based on the contour point coordinates; determining the point cloud corner point coordinates of the calibration board based on the contour point list; determining the image corner point coordinates based on each image data; and determining the transformation relationship between the radar and the image acquisition device based on the point cloud corner point coordinates and the image corner point coordinates. This method solves the problem of inaccuracy in joint calibration of camera and radar, and the calibration process is simple and the calibration results are accurate.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for calibrating radar and cameras and transforming coordinates. Background Technology

[0002] With the development of intelligent driving technology, more and more vehicles are equipped with radar and cameras to collect data and achieve intelligent driving. When collecting data through radar and cameras, it is necessary to convert the data collected by radar and cameras to the same coordinate system, therefore, radar and camera calibration is required.

[0003] In the joint calibration of radar and camera, due to the sparsity of radar point clouds, it is difficult to select the precise coordinates of the calibration target; different radar devices have different accuracies at local locations, and the joint calibration will be affected by equipment errors; existing joint calibration methods are complex and have low accuracy, and cannot effectively and quickly complete the joint calibration of camera and radar. Summary of the Invention

[0004] This invention provides a method for calibrating radar and camera and transforming coordinates to solve the problem of low accuracy in radar and camera calibration.

[0005] According to one aspect of the present invention, a method for calibrating radar and camera is provided, comprising:

[0006] Acquire at least one frame of point cloud data and image data corresponding to each of the point cloud data, wherein the point cloud data is obtained by a radar acquisition calibration board and the image data is obtained by an image acquisition device acquisition calibration board;

[0007] For each frame of point cloud data, the point cloud data is processed to obtain the point cloud data corresponding to the calibration board. The point cloud data corresponding to the calibration board is translated and rotated to a preset plane to obtain the corresponding first point cloud data.

[0008] Cluster the first point cloud data to obtain at least one cluster, determine the contour point coordinates of each cluster based on the second point cloud data in each cluster, determine the contour point list based on the contour point coordinates, and determine the point cloud corner point coordinates of the board based on the contour point list.

[0009] The corner coordinates of the images are determined based on the image data, and the conversion relationship between the radar and the image acquisition device is determined based on the corner coordinates of the point cloud and the image corner coordinates.

[0010] According to another aspect of the present invention, a coordinate transformation method for radar and camera is provided, comprising:

[0011] Acquire point cloud data to be converted, which is collected by radar;

[0012] The point cloud data to be converted is subjected to coordinate transformation according to the conversion relationship between radar and image acquisition device to obtain target data. The conversion relationship between radar and image acquisition device is determined by the radar and camera coordinate calibration method according to any embodiment of the present invention.

[0013] According to another aspect of the present invention, a radar and camera calibration device is provided, comprising:

[0014] The data acquisition module is used to acquire at least one frame of point cloud data and image data corresponding to each of the point cloud data. The point cloud data is obtained by a radar acquisition calibration board, and the image data is obtained by an image acquisition device acquisition calibration board.

[0015] The first point cloud data determination module is used to process the point cloud data for each frame of point cloud data to obtain the point cloud data corresponding to the calibration board, and to translate and rotate the point cloud data corresponding to the calibration board to a preset plane to obtain the corresponding first point cloud data.

[0016] The corner coordinate determination module is used to cluster each of the first point cloud data to obtain at least one cluster, determine the contour point coordinates of each cluster based on the second point cloud data in each cluster, determine the contour point list based on the contour point coordinates, and determine the point cloud corner coordinates of the board based on the contour point list.

[0017] The conversion relationship determination module is used to determine the corner coordinates of the image based on the image data, and to determine the conversion relationship between the radar and the image acquisition device based on the corner coordinates of the point cloud and the image corner coordinates.

[0018] According to another aspect of the present invention, a coordinate transformation device for radar and camera is provided, comprising:

[0019] The data to be converted acquisition module is used to acquire point cloud data to be converted, which is collected by radar.

[0020] The coordinate transformation module is used to perform coordinate transformation on the point cloud data to be transformed according to the transformation relationship between the radar and the image acquisition device to obtain target data. The transformation relationship between the radar and the image acquisition device is determined by the radar and camera coordinate calibration method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0022] At least one processor; and

[0023] A memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method described in any embodiment of the present invention.

[0025] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the method described in any embodiment of the present invention.

[0026] The technical solution of this invention involves acquiring at least one frame of point cloud data and corresponding image data for each point cloud data. The point cloud data is obtained by a radar calibration board, and the image data is obtained by an image acquisition device from the calibration board. For each frame of point cloud data, the point cloud data is processed to obtain point cloud data corresponding to the calibration board. The point cloud data corresponding to the calibration board is then translated and rotated to a preset plane to obtain corresponding first point cloud data. Each first point cloud data is clustered to obtain at least one cluster. The contour point coordinates of each cluster are determined based on the second point cloud data in each cluster. A contour point list is determined based on the contour point coordinates. The point cloud corner coordinates of the calibration board are determined based on the contour point list. The image corner coordinates are determined based on each image data. The conversion relationship between the radar and the image acquisition device is determined based on the point cloud corner coordinates and the image corner coordinates. This solves the problem of phase... To address the inaccuracy issue in joint calibration of the radar and camera, after acquiring point cloud data and its corresponding image data, the point cloud data is first processed to filter out invalid point clouds and retain valid point clouds, resulting in point cloud data corresponding to the calibration board. Then, the point cloud data corresponding to the calibration board is moved to a preset plane through rotation and translation operations to obtain the first point cloud data. The first point cloud data is then clustered to obtain clusters. The coordinates of the contour points of each cluster are determined based on the second point cloud data within each cluster. A contour point list is formed based on the contour point coordinates, and the coordinates of the point cloud corner points of the calibration board are determined based on the contour point list, automatically determining the coordinates of the corner points of the calibration board with accurate results. The coordinates of the image corner points are determined through image data. Finally, the conversion relationship between the radar and the image acquisition device is determined based on the coordinates of each point cloud corner point and the image corner point coordinates, completing the calibration of the radar and camera. The calibration method provided in this application embodiment has a simple calibration process and accurate calibration results.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart of a radar and camera calibration method according to Embodiment 1 of the present invention;

[0030] Figure 2 This is a flowchart of a radar and camera calibration method according to Embodiment 2 of the present invention;

[0031] Figure 3 This is a flowchart of a radar and camera calibration method according to Embodiment 3 of the present invention;

[0032] Figure 4a This is a schematic diagram illustrating a cluster according to Embodiment 3 of the present invention;

[0033] Figure 4b This is an example diagram showing each contour point in a contour point list according to Embodiment 3 of the present invention;

[0034] Figure 4c This is a display example diagram of a corner point provided according to Embodiment 3 of the present invention;

[0035] Figure 5 This is a flowchart of a radar-camera coordinate transformation method according to Embodiment 4 of the present invention;

[0036] Figure 6 This is a schematic diagram of the structure of a radar and camera calibration device according to Embodiment 5 of the present invention;

[0037] Figure 7 This is a schematic diagram of the structure of a radar-camera coordinate transformation device according to Embodiment Six of the present invention;

[0038] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 7 of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] Example 1

[0042] Figure 1 This is a flowchart illustrating a radar and camera calibration method according to Embodiment 1 of the present invention. This embodiment is applicable to the joint calibration of radar and camera. The method can be executed by a radar and camera calibration device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0043] S101. Acquire at least one frame of point cloud data and the corresponding image data for each point cloud data. The point cloud data is obtained from the radar acquisition calibration board, and the image data is obtained from the image acquisition device acquisition calibration board.

[0044] In this embodiment, the radar can be a lidar, millimeter-wave radar, ultrasonic radar, etc.; the image acquisition device can be a camera, video recorder, etc. The radar and image acquisition device need to maintain a constant relative position during calibration. Therefore, a mechanical structure can be used to fix the radar and image acquisition device to maintain their spatial relative position. During calibration, the radar and image acquisition device are first fixed. Then, a calibration board is placed within the acquisition range of the radar and image acquisition device. The radar and image acquisition device are controlled to capture point cloud data and image data. The radar and image acquisition device can be controlled to acquire multiple point cloud data and image data of the calibration board at the same location at a certain frequency. The calibration board is moved to acquire point cloud data and image data of the calibration board at other locations. When moving the calibration board, it can be controlled to move from near to far (i.e., further and further away from the radar and image acquisition device), ensuring that the calibration board is evenly distributed throughout the space as much as possible, and maintaining the calibration board within the complete visual range of the radar and image acquisition device (i.e., multiple locations of the calibration board during movement are evenly distributed within the space that the image acquisition device and radar can capture). In this embodiment of the application, when the radar and image acquisition device acquires point cloud data and image data, it can obtain real-time data using the software development kit (SDK) provided by the manufacturer of the radar and image acquisition device.

[0045] The point cloud data and corresponding image data acquired in this step can be one location on the calibration board for each frame of data, or multiple frames of data for each location. This embodiment can directly acquire at least one frame of point cloud data collected by the radar, as well as the corresponding image data for each frame of point cloud data. The image data corresponding to the point cloud data can be determined based on the acquisition time of the point cloud data. For example, image data and point cloud data acquired at the same time are corresponding data, or acquisition time differences within a certain range are corresponding data, and so on.

[0046] S102. For each frame of point cloud data, process the point cloud data to obtain the point cloud data corresponding to the calibration board, and translate and rotate the point cloud data corresponding to the calibration board to a preset plane to obtain the corresponding first point cloud data.

[0047] In this embodiment, the preset plane can be a pre-determined plane, such as the xoy plane; the first point cloud data can be specifically understood as the coordinate data of each point forming the calibration plate on the preset plane.

[0048] For each frame of point cloud data, the coordinates of the corner points of the acquired calibration board can be determined using steps S102-S103. The point cloud data is then processed to filter out invalid data, retaining only the point cloud data corresponding to the calibration board. Filtering can be performed using a filter, applying preset rules and range parameters to obtain the point cloud data that forms the calibration board, i.e., the point cloud data corresponding to the calibration board. The point cloud data corresponding to the calibration board is then translated and rotated to a preset plane. This transformation can be achieved by multiplying the point cloud data by a transformation matrix, which can be determined based on the plane containing the point cloud data corresponding to the calibration board.

[0049] S103. Cluster the first point cloud data to obtain at least one cluster. Determine the contour point coordinates of each cluster based on the second point cloud data in each cluster. Determine the contour point list based on the contour point coordinates. Determine the point cloud corner point coordinates of the board based on the contour point list.

[0050] In this embodiment, the second point cloud data can be specifically understood as the point cloud data in the cluster; the contour point coordinates can be understood as the coordinates of the contour points of the calibration board; the contour point list can be specifically understood as a list composed of the points of the contour of one side of the calibration board; the point cloud corner point coordinates can be specifically understood as the point cloud coordinates of the corner points of the calibration board, that is, the coordinates of the corner points collected by the radar. The corner points are the vertices of the calibration board. The number of corner points depends on the shape of the calibration board. Taking the calibration board as a rectangle as an example, the number of corner points is 4.

[0051] Specifically, the first point cloud data is clustered according to a pre-determined clustering algorithm to filter out outliers, resulting in at least one cluster. The clustering algorithm can be the DBSCAN clustering algorithm. The point cloud data included in each cluster is the second point cloud data. For each cluster, the horizontal and vertical coordinates of the second point cloud data in the cluster are compared, or the coordinates of the contour points are determined by comparing the distances between points. The number of contour point coordinates in each cluster is at least two. The coordinates of each contour point are counted, and the contour points that constitute an edge are determined according to preset rules, algorithms, and other methods. A contour point list is formed based on the contour point coordinates of this part of the contour points. In this embodiment, the number of contour point lists depends on the shape of the calibration board. The number of contour point lists corresponds to the number of edges of the calibration board. The edges of the calibration board are determined based on the contour points in the contour point list, and the coordinates of the corner points of the point cloud of the calibration board are determined based on the intersections of the edges.

[0052] S104. Determine the corner coordinates of the images based on the image data, and determine the conversion relationship between the radar and the image acquisition device based on the corner coordinates of the point cloud and the image corner coordinates.

[0053] In this embodiment, the image corner coordinates can be specifically understood as the coordinates of the corner points of the calibration board acquired by the image acquisition device. The image corner coordinates are coordinates in the coordinate system of the image acquisition device. The conversion relationship between the radar and the image acquisition device can be a conversion matrix, which is used to convert the point cloud data acquired by the radar and the data acquired by the image acquisition device to the same coordinate system.

[0054] Specifically, image data is processed to determine the corner coordinates of each corner point in the image data. This can be achieved through algorithms, neural network models, or by manually selecting the corner coordinates. For example, user actions such as single-clicking or double-clicking can be used to determine the coordinates of the selected point, which is then used as the image corner coordinates. For each set of corresponding point cloud data and image data, since they were captured from the same calibration board at the same location, their coordinates should be consistent in world space. Therefore, there is a certain transformation relationship between the two coordinate systems. A transformation relationship matrix is ​​calculated based on the corner coordinates of each point cloud and image, yielding the transformation relationship between the radar and the image acquisition device. This transformation relationship matrix can be further optimized to obtain the final transformation relationship between the radar and the image acquisition device, a method with higher accuracy.

[0055] This invention provides a calibration method for radar and camera, solving the problem of inaccuracy in joint calibration of camera and radar. After acquiring point cloud data and its corresponding image data, the point cloud data is first processed to filter out invalid point clouds and retain valid point clouds, obtaining the point cloud data corresponding to the calibration board. The point cloud data corresponding to the calibration board is then moved to a preset plane through rotation and translation operations to obtain the first point cloud data. The first point cloud data is clustered to obtain clusters, and the contour point coordinates of each cluster are determined based on the second point cloud data in each cluster. A contour point list is formed based on the contour point coordinates, and then the point cloud corner point coordinates of the calibration board are determined based on the contour point list, automatically determining the coordinates of the corner points of the calibration board with accurate results. The image corner point coordinates are determined through image data, and finally, the conversion relationship between the radar and the image acquisition device is determined based on the corner point coordinates of each point cloud and the image corner point coordinates, completing the calibration of radar and camera. The calibration method provided in this application embodiment has a simple calibration process and accurate calibration results.

[0056] Example 2

[0057] Figure 2 This is a flowchart of a radar and camera calibration method provided in Embodiment 2 of the present invention. This embodiment is a refinement based on the above embodiments. Figure 2 As shown, the method includes:

[0058] S201. Acquire at least one frame of point cloud data and the corresponding image data for each point cloud data. The point cloud data is obtained from the radar acquisition calibration board, and the image data is obtained from the image acquisition device acquisition calibration board.

[0059] As an optional embodiment of this example, this optional embodiment further optimizes the acquisition of image data corresponding to each point cloud data as follows:

[0060] A1. For each frame of point cloud data, determine the timestamp corresponding to the point cloud data.

[0061] After acquiring point cloud data, the radar saves the data, simultaneously adding a timestamp. This timestamp can be stored in the relevant attribute information of the point cloud data, or in the file name corresponding to the point cloud data, etc. The timestamp indicates the acquisition time of the point cloud data. During calibration, the device retrieves the point cloud data from its corresponding storage space, along with its timestamp.

[0062] A2. Filter out the image data corresponding to the point cloud data based on the timestamp.

[0063] Similarly, after acquiring image data, the image acquisition device saves the image data and simultaneously timestamps it. The timestamp can be stored in the image data's relevant attribute information, or in the file name corresponding to the image data, etc. The timestamp indicates the acquisition time of the image data. During calibration, this execution device can retrieve all image data from its storage space and then filter it based on the timestamp to determine the image data corresponding to the point cloud data. Alternatively, it can directly filter the image data corresponding to the point cloud data from its storage space based on the timestamp. One method of filtering image data based on timestamps is to determine the image data whose time is closest to that of the point cloud data, and use this image data as the corresponding image data for the point cloud data.

[0064] Taking LiDAR as an example, the image acquisition device has a higher frequency than LiDAR, but lower real-time performance. Therefore, by marking timestamps, the most recent camera frame can be selected as the reference for LiDAR data to acquire data.

[0065] S202. For each frame of point cloud data, filter the point cloud data according to the point cloud range corresponding to the calibration board to determine candidate point clouds.

[0066] In this embodiment, the candidate point cloud can be understood as the point cloud data obtained after preliminary filtering of the point cloud data. The point cloud data consists of multiple (x, y, z) points, in mm, with the direction based on the positive direction of the lidar. After acquiring the point cloud data, it is necessary to filter out the point cloud related to the calibration board. The point cloud data can be preset, or the point cloud range can be manually input or selected. For example, the pass-through filter provided by the PCL point cloud processing library can be used to initially limit the point cloud range. The maximum and minimum values ​​of the coordinates of the three axes (x, y, z) can be selected by sliding the slider in OpenCV to obtain the candidate point cloud. The point cloud after the range limitation can be adjusted and displayed in real time through the above method.

[0067] S203. Filter the candidate point clouds to determine the target point cloud.

[0068] In this embodiment, the target point cloud can be specifically understood as the point cloud of the calibration board obtained after further filtering of the point cloud. Due to the error of the radar receiver and related reasons such as light wave interference, there are still a few noisy points near the initially screened candidate point clouds. Therefore, the candidate point clouds are further filtered. The filtering method can be pre-selected, and the target point cloud is obtained by filtering out invalid points.

[0069] For example, a statistical filter can be used to filter candidate point clouds. The statistical filter performs a statistical analysis on the neighborhood of each point, obtaining the average distance from that point to all its neighbors. Assuming the final data for all points follows a Gaussian distribution, the shape of which is determined by the mean and standard deviation, points whose average distance falls outside the standard range (defined by the mean and standard deviation) are considered outliers. Since the calibration board is a single entity, the fluctuating points are scattered around it; these are considered outliers and removed using a statistical filter. In the figure, green points represent points within the plane, while the remaining colored points represent points outside the plane.

[0070] S204. Perform plane fitting based on the target point cloud, determine the plane equation of the fitting plane, and select the point cloud in the plane based on the plane equation.

[0071] In this embodiment, the in-plane point cloud refers to the target point cloud located within the plane of the fitted plane. A plane fitting algorithm is pre-set, and the target point cloud is fitted using the plane fitting algorithm to obtain the plane equation ax + by + cz + d = 0. The target point cloud is then filtered based on the range corresponding to the plane equation, removing points that exceed the fitted plane, and further filtering out the in-plane point cloud.

[0072] S205. Project the point cloud in the plane onto the fitting plane to obtain the point cloud data corresponding to the calibration plate.

[0073] Since the target point cloud may actually be on different planes, in order to facilitate subsequent translation, rotation and other processing, it is necessary to reproject the point cloud in the plane onto the fitting plane. This can be done by creating a projector to project the point cloud in the plane and obtain the point cloud data corresponding to the calibration board, which are in the same three-dimensional space plane.

[0074] S206. Select the reference point from the point cloud data corresponding to the calibration board, and subtract the coordinates of the reference point from the point cloud data corresponding to the calibration board to obtain the translated point cloud data.

[0075] Steps S206-S208 are used to translate and rotate the point cloud data of the calibration board to achieve dimensionality reduction of the point cloud data.

[0076] In this embodiment, the reference point can be specifically understood as a coordinate point used as a reference during translation; the reference point coordinates can be specifically understood as the coordinates of the reference point. A point cloud data point is randomly selected from the point cloud data corresponding to the calibration board as the reference point, and the coordinates of the reference point can be determined accordingly. The coordinates of the reference point are subtracted from the coordinates of all other point cloud data corresponding to the calibration board (the x, y, and z coordinates are all subtracted), and the resulting data is the translated point cloud data.

[0077] For example, based on the known equation of the fitted plane, ax + by + cz + d = 0, the equation of the normal line passing through the origin is obtained as cz + d = 0. Taking any point (x0, y0, z0) on the original plane as the reference point, subtracting the coordinates of this point from all points in the fitted plane yields the translated point cloud data and the new plane equation ax + by + cz = 0. This achieves the translation of the calibration plate plane to the plane where the origin is located, and the normal line of the translated plane intersects the x-axis and y-axis at the origin.

[0078] S207. Determine the first rotation matrix and the second rotation matrix based on the plane equation of the fitted plane.

[0079] In this embodiment, both the first rotation matrix and the second rotation matrix are matrices used to rotate from one plane to another. The expressions for the first and second rotation matrices are predetermined, and both expressions include unknown parameters, typically angles. A relationship is established between the rotation matrix and the normal vectors of the planes before and after rotation. The calculation formula for the parameters of the rotation matrix is ​​determined through computation. The parameters of the plane equation of the fitted plane are substituted into the calculation formula for the parameters of the rotation matrix to obtain the parameters of the rotation matrix, thus obtaining the rotation matrix, i.e., the first rotation matrix and the second rotation matrix.

[0080] As an optional embodiment of this example, the first rotation matrix and the second rotation matrix are further determined based on the plane equation of the fitted plane, optimized as follows:

[0081] B1. Determine the plane normal vector based on the plane equation of the fitted plane, determine the first rotation parameter based on the plane normal vector, and determine the first rotation matrix based on the first rotation parameter.

[0082] In this embodiment, the first rotation parameter can be specifically understood as a parameter in the first rotation matrix, typically an angle. The calculation formula for the first rotation parameter is predetermined. The parameters in the calculation formula include the coordinates of the plane normal vector. Therefore, after determining the plane normal vector, substituting it into the calculation formula yields the first rotation parameter. Substituting the first rotation parameter into the expression of the first rotation matrix yields the first rotation matrix.

[0083] For example, this application provides a method for determining a first rotation matrix:

[0084] According to the definition of the dot product of vectors, the normal vector of the original fitted plane is (a,b,c). To make the normal vector of the rotated plane lie on the xz plane, we have (a',b',c')·(0,1,0)=0, where (a',b',c') is the normal vector after rotation.

[0085] The expression for the first rotation matrix is: The first rotation parameter is θ;

[0086] Substitute the plane normal vector of the fitted plane into the following formula:

[0087]

[0088] We get acosθ + csinθ = 0, and solve for θ to get -arctan(a / c);

[0089] Therefore, given the plane normal vector, the first rotation parameter θ can be determined, and the first rotation matrix can be determined based on the first rotation parameter.

[0090] B2. Determine the normal vector of the rotated plane based on the first rotation parameter, determine the second rotation parameter based on the normal vector of the rotated plane, and determine the second rotation matrix based on the second rotation parameter.

[0091] Substitute the first rotation parameter and the plane normal vector of the fitted plane into the above formula to calculate the normal vector (a', b', c') of the rotated plane. Substitute the normal vector of the rotated plane into the formula for calculating the second rotation parameter to obtain the second rotation parameter. Substitute the second rotation parameter into the expression for the second rotation matrix to obtain the second rotation matrix.

[0092] For example, the expression for the second rotation matrix is: φ is the second rotation parameter;

[0093] The formula for calculating the second rotation parameter is as follows:

[0094]

[0095] Solving for φ, we get φ = -arctan(b′ / c′). Given the normal vector (a′, b′, c′) of the rotated plane, we substitute it into the formula to calculate φ. Substituting φ into the expression for the second rotation matrix, we obtain the second rotation matrix. This method can rotate a plane to the xoy plane.

[0096] S208. Based on the first rotation matrix and the second rotation matrix, rotate the translated point cloud data to a preset plane to obtain the corresponding first point cloud data.

[0097] Multiply the first rotation matrix by the translated point cloud data to achieve the first rotation, obtaining the coordinates after the first rotation. Then, multiply the stacked rotation matrix by the coordinates after the first rotation to obtain the coordinates of the first point cloud data. Taking the above rotation matrix calculation process as an example, the preset plane where the first point cloud data is located is the xoy plane. Alternatively, the normal vector of other planes can be selected for calculation to translate and rotate the point cloud data to other planes.

[0098] For example, taking the coordinates of the translated point cloud data as (x', y', z'), this point can be rotated to the xoy plane in the following way.

[0099]

[0100]

[0101] Where (x1,y1,z1) are the point cloud coordinates after the first rotation; (x2,y2,z2) are the point cloud coordinates after the second rotation, which are the coordinates of the preset plane. When the preset plane is the xoy plane, z2 = 0.

[0102] S209. Cluster the first point cloud data to obtain at least one cluster. Determine the contour point coordinates of each cluster based on the second point cloud data in each cluster. Determine the contour point list based on the contour point coordinates. Determine the point cloud corner point coordinates of the board based on the contour point list.

[0103] S210. Determine the corner coordinates of the images based on the image data, and determine the conversion relationship between the radar and the image acquisition device based on the corner coordinates of the point cloud and the image corner coordinates.

[0104] When determining the coordinates of image corner points, you can determine the coordinates of image corner points sequentially according to the arrangement order of the point cloud corner point coordinates, or you can manually select the corner points in the image in sequence to determine the coordinates of the image corner points.

[0105] After obtaining the corner coordinates of point clouds and images from multiple frames, camera intrinsic parameters can be obtained through single-target positioning. This application takes the transformation of radar-acquired point cloud data into an image coordinate system as an example, solving for the 3D-to-2D RT matrix, which can be optimized using the least squares method.

[0106] Assume there are n corner points on the calibration plate, and the coordinates of these corner points in the point cloud acquired by the radar are (Xi, Yi, Zi), where the radar-acquired corner point coordinates are in the world coordinate system; the coordinates of the corner points in the image acquired by the image acquisition device are (ui, vi). Then, from 3D to...

[0107]

[0108] The transformation matrix for 2D is as follows:

[0109] Among them, f x and f y It refers to the camera's focal length in both the horizontal and vertical directions, c x ,c y It is the position of the origin of the pixel coordinate system, r ij and t i These are the parameters contained in the rotation matrix and translation vector.

[0110] Representing the transformation matrix as a vector, we get:

[0111]

[0112] Let the above expression be Ax = b, then the least squares solution is x = (A T A) -1 A T b. The rotation matrix R and the translation vector t can be obtained from the above equation. Wherein,

[0113] To ensure that the solved matrix satisfies the requirements of rotation matrix and translation vector, the SVD decomposition method is used to correct the matrix. The solved rotation matrix R is decomposed using SVD to obtain R = USV. T Where U and V are orthogonal matrices, and S is a diagonal matrix. Then R is reconstructed as R = UV. T Then, divide the translation vector t by σ1, i.e., t' = t / σ1. The resulting rotation matrix R and translation vector t' represent the required transformation relationship between the radar and the image acquisition device.

[0114] This invention provides a calibration method for radar and camera, solving the problem of inaccuracy in joint camera and radar calibration. After acquiring point cloud data and its corresponding image data, the point cloud data is first processed by filtering out invalid point clouds through two filtering operations, retaining valid point clouds to obtain the point cloud data corresponding to the calibration board, thus improving the accuracy of the point cloud data. The point cloud data is then translated by selecting reference points, and a first rotation matrix and a second rotation matrix are determined based on the plane equation of the fitted plane. The point cloud data is then rotated based on these first and second rotation matrices, rotating the point cloud data corresponding to the calibration board to a preset plane to obtain the first point cloud data. This facilitates subsequent calculation of the corner coordinates of the calibration board and improves the accuracy of corner coordinate calculation. The first point cloud data is processed to determine the corner coordinates of the calibration board, automatically determining the corner coordinates of the calibration board with accurate results. Image corner coordinates are determined through image data. Finally, the conversion relationship between the radar and the image acquisition device is determined based on the corner coordinates of each point cloud and the image corner coordinates, completing the radar and camera calibration. The calibration method provided in this application is simple in process and accurate in results.

[0115] Example 3

[0116] Figure 3 This is a flowchart of a radar and camera calibration method provided in Embodiment 3 of the present invention. This embodiment is a refinement based on the above embodiments. Figure 3 As shown, the method includes:

[0117] S301. Acquire at least one frame of point cloud data and the corresponding image data for each point cloud data. The point cloud data is obtained from the radar acquisition calibration board, and the image data is obtained from the image acquisition device acquisition calibration board.

[0118] S302. For each frame of point cloud data, process the point cloud data to obtain the point cloud data corresponding to the calibration board, and translate and rotate the point cloud data corresponding to the calibration board to a preset plane to obtain the corresponding first point cloud data.

[0119] S303. Cluster the first point cloud data to obtain at least one cluster.

[0120] Taking the clustering of the first point cloud data using the DBSCAN clustering algorithm as an example, the derivation of the DBSCAN clustering algorithm formula for clustering the two-dimensional first point cloud data (ignoring z-coordinates that are all 0) is as follows:

[0121] a. Define the distance function d(i,j)

[0122]

[0123] Where, x i ,yi Let x represent the x and y coordinates of point i. j ,y j This represents the x-coordinate and y-coordinate of point j.

[0124] b. Find the ε-neighborhood N of each point. ε (i)

[0125] N ε (i)={j:d(i,j)≤ε}

[0126] Here, ε is a parameter of the DBSCAN algorithm, representing the radius centered at point i. For each sample point i, all points whose distance is less than or equal to ε are placed in its ε-neighborhood.

[0127] c. Types of dividing points: core points, boundary points, noise points

[0128] For each sample point i, it is classified into one of the following three types based on the number of points in its ε-neighborhood:

[0129] Key point: If N ε If there are at least minpts points in (i) (minpts is another parameter of the DBSCAN algorithm, representing the minimum number of sample points required in a cluster), then point i is the core point.

[0130] Boundary point: If N ε If there are fewer than minpts points in (i), but there exists a core point j such that point i is in the ε-neighborhood of j, then point i is a boundary point.

[0131] Noise point: If point i is neither a core point nor a boundary point, then point i is a noise point.

[0132] d. Constructing clusters

[0133] For each core point i, merge it with all core points within its ε-neighborhood into the same cluster. Simultaneously, add all boundary points adjacent to these core points to that cluster. Repeat this process until all core points have been visited. If some points cannot be assigned to any cluster, they are marked as noise points.

[0134] The algorithm flow is as follows:

[0135] 1. For each sample point i(x,y), find its ε-neighborhood N ε (i).

[0136] 2. If N ε If there are at least minpts points in (i), then point i is marked as the core point, and N is set to... ε All points in (i) are assigned to the cluster containing i.

[0137] 3. For all unassigned points, mark them as noise points or boundary points. If point i is in the ε-neighborhood of some core point j, assign it to the cluster where j belongs; otherwise, mark it as a noise point.

[0138] 4. Repeat step 3 until all points are assigned to a cluster or marked as noise points.

[0139] By using the above method for clustering, at least one cluster can be obtained.

[0140] S304. For each cluster, determine the center point based on the second point cloud data in the cluster.

[0141] For each cluster, the coordinates of the cluster's outline points are determined using steps S304-S207. For each cluster, all the second point cloud data contained in the cluster are determined, and the center point is calculated based on the coordinate values ​​of the second point cloud data. For example, the average value of the coordinates of each point is calculated, and the resulting value is the center point.

[0142] S305. Calculate the first distance between each second point cloud data and the center point, and determine the second point cloud data corresponding to the first distance with the farthest first distance as the first contour point.

[0143] In this embodiment, the first distance can be specifically understood as the distance between the second point cloud data and the center point. The distance between each second point cloud data in the cluster and the center point is calculated according to the distance calculation formula to obtain the first distance; the first distances are compared to determine the maximum value, that is, the first distance with the farthest distance, and the second point cloud data corresponding to this first distance is determined as the first contour point.

[0144] S306. Calculate the second distance between each second point cloud data and the first contour point, and determine the second point cloud data corresponding to the second distance with the farthest distance as the second contour point.

[0145] In this embodiment, the second distance can be specifically understood as the distance between the second point cloud data and the first contour point. Taking the first contour point as the starting point, the distance between each second point cloud data in the cluster and the first contour point is calculated according to the distance calculation formula to obtain the second distance; the second distances are compared, and the maximum value is determined, that is, the second distance with the farthest distance, and the second point cloud data corresponding to this second distance is determined as the second contour point.

[0146] S307. Determine the coordinates of the cluster's contour points based on the coordinates of the first contour point and the second contour point.

[0147] The coordinates of the first contour point and the second contour point are the contour point coordinates of the cluster.

[0148] S308, Starting from a contour point in the first cluster.

[0149] The starting point is selected from one of the contour points in the first cluster. This selection can be random or based on certain rules, such as choosing the contour point with the smaller x-coordinate. The clusters in this embodiment can be sorted according to certain rules, such as from left to right.

[0150] S309. Calculate the distance between the contour point in the next cluster and the starting point, take the closest contour point as the target contour point, and determine the straight line direction between the target contour point and the starting point.

[0151] In this embodiment, the target contour point can be specifically understood as the contour point used to determine the contour point list. The contour points in the next cluster of the first cluster are determined, and the distances between each two contour points and the starting point are calculated. The calculated distances are compared, and the contour point with the closest distance is taken as the target contour point. The direction of the line is determined by calculating the angle between the target contour point and the starting point, the slope of the line, etc. For example, if the coordinates of the target contour point are (3,4) and the coordinates of the starting point are (2,3), the direction of the line can be determined to be that the starting point points to the upper right.

[0152] S310. If the direction of the straight line changes and the number of all target contour points meets the requirements, generate a list of contour points based on the contour points corresponding to the starting point and the target contour points with the same direction of the straight line. Take the latest target contour point as the new starting point and return to the execution of step S309.

[0153] After determining the direction of a straight line, it is checked whether the direction has changed. The first straight line direction is assumed to be unchanged since there is no previous straight line direction. After determining two or more straight line directions, it is checked whether the direction has changed based on the previous straight line direction. If the direction has changed compared to the previously determined direction, it is checked whether the cumulative number of target contour points meets the requirements. The number of target contour points meets the requirements if it is not less than a certain threshold and is less than the total number of clusters. The threshold can be determined based on the total number of clusters, for example, half the total number of clusters. If the requirements are met, the contour point corresponding to the first starting point is determined, as well as the target contour points with the same straight line direction (i.e., all target contour points except the last one). A contour point list is generated based on the determined contour points and the target contour points.

[0154] The latest target contour point determined at this point is used as the new starting point, and the process returns to step S309 to determine the direction of the line. Since multiple target contour points may have been determined during the method execution, the latest target contour point (i.e., the last determined target contour point) is used as the new starting point. If only one target contour point is determined, this point is directly used as the new starting point. This application can also accumulate the number of target contour points after they are determined for subsequent judgment.

[0155] After generating a list of contour points, the last target contour point obtained at this time is taken as the new starting point, and the process returns to step S309 to recalculate the direction and determine a new list of contour points. In this embodiment, once the contour point list is formed, the contour points in this list no longer need to participate in the calculation, and the number of target contour points is re-accumulated. This embodiment can also determine the position of the latest target contour point's cluster within all clusters. If the straight-line direction changes and the position of the latest target contour point's cluster meets the requirements, a list of contour points is generated based on the contour point corresponding to the starting point and each target contour point. For example, if the latest target contour point's cluster is located in the middle position among all clusters, considering the shape of the calibration plate and whether the total number of clusters is odd or even, this middle position can be an approximate position, not necessarily the absolute middle.

[0156] S311. If the direction of the straight line changes and the number of target contour points does not meet the requirements, filter out the target contour points, take the latest unfiltered target contour point as the new starting point, and return to execute step S309.

[0157] If the direction of the straight line changes and the number of target contour points does not meet the requirements, it can be considered as an error. This target contour point will be filtered out, not added to the contour point list, and will not be included in the calculation of corner coordinates.

[0158] S312. If the direction of the straight line has not changed and the target contour point is not a contour point in the last cluster, take the latest target contour point as the new starting point and return to execute step S309.

[0159] If the direction of the straight line has not changed, determine whether the target contour point is in the last cluster. If not, take the latest target contour point as the new starting point and return to execute step S309.

[0160] S313. If the direction of the straight line has not changed and the target contour point is the contour point in the last cluster, generate a contour point list based on the contour point corresponding to the starting point and each target contour point.

[0161] S314. Determine whether all contour points have been traversed. If not, execute S315; if yes, execute S316.

[0162] If the direction of the straight line has not changed, and the target contour point is the contour point in the last cluster, it can be determined that all the points constituting one edge of the calibration plate have been determined. A list of contour points is generated based on the starting point and the target contour points with the same straight line direction after the starting point.

[0163] It should be noted that when generating a list of contour points, this application uses the first starting point, the last target contour point aligned with the straight line direction, and other target contour points in between to form the contour point list. After forming a contour point list, a new first starting point and a new last target contour point aligned with the straight line direction are determined.

[0164] After traversing the contour points in the last cluster, determine whether all contour points in all clusters have been traversed. If yes, execute S316, at which point the list of all contour points has been generated. If not, execute S315.

[0165] S315. Using another contour point in the first cluster as the new starting point, return to the execution of step S309.

[0166] S316. Ensure that all contour point lists have been generated.

[0167] If not all contour points have been traversed, since the last cluster has already been traversed, we can return to the first cluster to start traversing again, using another contour point in the first cluster as the new starting point, and repeat step S309 to determine the direction of the straight line in order to generate another list of contour points.

[0168] For example, Figure 4aAn example diagram illustrating clusters is provided, with each cluster containing two contour points. The process of generating the contour point list is illustrated below. In the diagram, the black dots represent contour points, and the rectangles represent a cluster, with each cluster containing two contour points. Using contour point 4011 in the first cluster 401 as the starting point, the distances between contour points 4021 and 4022 in the second cluster 402 (i.e., the next cluster) and contour point 4011 are calculated. The closest contour point 4021 is taken as the target contour point. The straight line direction between the target contour point and the starting point is determined to be the upper right. In this embodiment, the straight line direction is taken as the direction from the starting point to the target contour point. At this time, there is only one straight line direction, and it is assumed that the straight line direction has not changed. The target contour point (i.e., contour point 4021) is taken as the new starting point, and the distances between it and contour points 4031 and 4032 in the third cluster 403 are calculated. The closest contour point 4031 is taken as the new target contour point. The straight line direction between this target contour point (contour point 4031) and the starting point (contour point 4021) is determined to be the upper right. At this time, it can be determined that the straight line direction has not changed, and the target contour point is not a contour point in the last cluster. The target contour point (contour point 4031) is taken as the new starting point, and the straight line direction is determined to be the upper right. Figure 4a As shown, when the starting point is contour point 4071, the corresponding target contour point is contour point 4081. At this time, the straight line direction is to the lower right, the straight line direction has changed, and the number of target contour points is 7 (from contour point 4021 to contour point 4081), which is roughly the same as half of the total number of clusters, thus meeting the requirements. Based on the first starting point (contour point 4011) and the target contour points (contour point 4021 to contour point 4071) before the straight line direction is changed, the first contour point list is formed by all the target contour points from contour point 4011 to contour point 4071. This contour point list stores the contour points of the upper left contour of the plate.

[0169] The last determined target contour point (contour point 4081) is used as the new starting point, and the straight line direction is recalculated until the contour point in the last cluster. The straight line direction has not changed. Since the target contour point is already the contour point in the last cluster (contour point 4131), the target contour points between contour point 4081 and contour point 4131 are formed into a second contour point list. This contour point list stores the contour points of the upper right contour of the plate.

[0170] At this point, not all contour points have been traversed. Instead, another contour point 4012 from the first cluster is used as the new starting point to recalculate the straight line direction. Figure 4aAs shown, the straight line direction between contour point 4012 and contour point 4042 is downward to the right. When calculating the straight line direction between contour point 4042 and the target contour point 4052, starting from contour point 4042, the straight line direction is upward to the right. The straight line direction has changed. However, at this time, the number of target contour points is 4 (from contour point 4022 to contour point 4052), which does not meet the requirements. Target contour point 4052 is filtered out. Starting from target contour point 4042, the straight line direction between contour point 4042 and target contour point 4052 is calculated. The direction of the straight line between 062 and the target contour point remains unchanged. Continuing to calculate the straight line direction between contour point 4062 and the target contour point 4072, the direction changes. The number of target contour points is now 6 (from contour point 4022 to contour point 4072), meeting the requirement. Based on contour points 4012, 4022, 4032, 4042, and 4062, a third contour point list is formed. This list stores the contour points of the lower right contour of the board.

[0171] Continue to use contour point 4072 as the new starting point to determine the direction of the straight line. Based on contour point 4072 to contour point 4132, a fourth contour point list is formed. At this time, all contour points have been traversed once, and it is confirmed that all contour point lists have been generated. This contour point list stores the contour points of the lower left contour of the plate.

[0172] The above steps can determine multiple contour point lists. Taking the calibration plate as a rectangle as an example, the number of contour point lists is 4.

[0173] S317. Perform line fitting on the contour points in each contour point list to determine the straight line equation of the contour line.

[0174] For each list of contour points, a straight line is fitted based on all contour points in the list. The straight line fitting method can be the least squares method, and the straight line equation of the contour line is obtained through fitting. In this embodiment, the list of contour points can be sorted in a top-bottom or left-right manner to obtain the straight line equations for the corresponding directions.

[0175] For example, the equation of the straight line is y = ax + b, where a is the slope and b is the intercept; the least squares formula is:

[0176]

[0177]

[0178] Where, x i and y i These are the x and y coordinates of a point on the line, and N is the number of contour points in the contour point list.

[0179] For each cluster, the contour points are arranged from left to right and from top to bottom. Then, the contour points are categorized, with the top-left, bottom-left, top-right, and bottom-right contour points placed in separate lists to form contour point lists. For each contour point list, a line fit can be performed using the least squares method to obtain the slope and intercept of a line, thus deriving the line equation.

[0180] S318. Calculate the intersection points based on the linear equations of each contour point to obtain the coordinates of the corner points.

[0181] Taking four contour points as an example (top left, top right, bottom right, and bottom left), the coordinates of each corner point can be calculated as follows:

[0182] 1. Top left corner: The intersection of the top left and top right lines is the coordinate of the top left corner.

[0183] 2. Top right corner point: The intersection of the top right and bottom right lines is the coordinate of the top right corner point;

[0184] 3. Bottom right corner point: The intersection of the bottom right and bottom left lines is the coordinate of the bottom right corner point;

[0185] 4. Bottom left corner point: The intersection of the bottom left and top left lines is the coordinate of the bottom left corner point.

[0186] For example, Figure 4b An example diagram is provided to display each contour point in the contour point list. The black dots in the diagram represent contour point 41. Only a few are marked as examples in the diagram. All other similar points are contour points 41. Figure 4c An example diagram illustrating corner points is provided, where each asterisk represents a corner point 42, and the diagram includes four corner points 42.

[0187] S319. Perform inverse transformation on the coordinates of each corner point based on the first rotation matrix and the second rotation matrix to obtain the transformed corner point coordinates.

[0188] Since the coordinate data used to calculate the corner coordinates in this embodiment of the application is data that has undergone translation and rotation processing, it is necessary to rotate and translate it back to the original plane after determining the corner coordinates.

[0189] The coordinates of each corner point are inversely transformed based on the first rotation matrix and the second rotation matrix. The first rotation matrix and the second rotation matrix are the rotation matrices used when the point cloud data corresponding to the calibration board is translated and rotated to the preset plane. The transformed corner point coordinates are obtained through two inverse rotations.

[0190] S320. Add the coordinates of the reference point to the coordinates of each transformed corner point to obtain the corner point coordinates of the point cloud.

[0191] The x, y, and z coordinates of each transformed corner point are added to the coordinates of the reference point to obtain the corner point coordinates of the point cloud. In the process of determining the corner point coordinates of the point cloud in this embodiment of the application, the point cloud data corresponding to the calibration board is first translated and rotated to place it in a preset plane, then clustered, and the corner point coordinates are obtained according to the resulting list of contour points. Finally, the obtained corner point coordinates are inversely transformed back to the original plane to obtain the corner point coordinates of the point cloud.

[0192] S321. Determine the corner coordinates of the images based on the image data, and determine the conversion relationship between the radar and the image acquisition device based on the corner coordinates of the point cloud and the image corner coordinates.

[0193] This invention provides a calibration method for radar and camera, solving the problem of inaccurate joint calibration of camera and radar. After obtaining the first point cloud data, the first point cloud data is first clustered to obtain clusters. Then, the distance is calculated based on the second point cloud data in each cluster to determine the contour point coordinates of each cluster. Then, the contour point coordinates are analyzed to automatically determine the contour point list. Then, a straight line is fitted based on the contour point list, and the corner coordinates of the calibration board are determined based on the intersection of the fitted straight line equations. Finally, the corner coordinates are inversely transformed to obtain the corner coordinates of the point cloud. The entire process of determining the corner coordinates of the calibration board is automated, saving manual resources, improving efficiency, and providing accurate results, which can reduce the impact of errors. Finally, the conversion relationship between radar and image acquisition device is determined based on the corner coordinates of each point cloud and the corner coordinates of the image, completing the calibration of radar and camera. The calibration method provided in this application is simple in calibration process and accurate in calibration results.

[0194] Example 4

[0195] Figure 5 This is a flowchart of a radar-camera coordinate transformation method provided in Embodiment 4 of the present invention. This embodiment is applicable to situations where coordinate transformation is performed between radar and camera. The method can be executed by a radar-camera coordinate transformation device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 5 As shown, the method includes:

[0196] S501. Acquire the point cloud data to be converted. The point cloud data to be converted is collected by radar.

[0197] In this embodiment, the point cloud data to be converted can be specifically understood as point cloud data requiring coordinate transformation; the point cloud data to be converted is acquired by radar, and the relative positional relationship between the radar and the image acquisition device remains unchanged, that is, the same as the relative position during the calibration process. If the relative position changes, recalibration is required.

[0198] The radar can collect point cloud data to be converted at a certain frequency. After each collection is completed, it can be transmitted to the execution device for real-time coordinate conversion; or it can be stored after collection and transmitted to the execution device in batches after the transmission conditions are met for unified processing; or it can be transmitted to the execution device after each collection is completed, and the execution device will process it uniformly after the processing conditions are met.

[0199] S502. Based on the conversion relationship between the radar and the image acquisition device, perform coordinate transformation on the point cloud data to be transformed to obtain target data. The conversion relationship between the radar and the image acquisition device is determined by the radar and camera coordinate calibration method according to any one of the embodiments of the present invention.

[0200] In this embodiment, the target data is the coordinates of the point cloud acquired by the radar in the coordinate system of the image acquisition device. The conversion relationship between the radar and the image acquisition device is determined in advance during the calibration process. The point cloud data to be converted is then transformed using the conversion relationship between the radar and the image acquisition device, and the resulting data is the target data.

[0201] Image acquisition devices can simultaneously acquire data with radar and perform data alignment, for example, determining the corresponding point cloud data and image data based on timestamps. After the target data is determined, the image data and target data are now in the same coordinate system, enabling obstacle recognition and intelligent driving control, among other things.

[0202] The radar and camera coordinate transformation method of this invention can transform the coordinates of point cloud data acquired by radar. By using the method provided in any embodiment of this application to determine the transformation relationship between radar and image acquisition device, the coordinate transformation of point cloud data is performed. Since the calibration result is accurate, accurate target data can be obtained when performing coordinate transformation. It can be applied in scenarios with high data accuracy, and the application scenarios are more extensive. When applied in the field of intelligent driving, it provides higher safety and a better user experience.

[0203] Example 5

[0204] Figure 6 This is a schematic diagram of a radar and camera calibration device provided in Embodiment 5 of the present invention. Figure 6 As shown, the device includes: a data acquisition module 61, a first point cloud data determination module 62, a corner coordinate determination module 63, and a transformation relationship determination module 64;

[0205] The data acquisition module 61 is used to acquire at least one frame of point cloud data and image data corresponding to each of the point cloud data. The point cloud data is obtained by a radar acquisition calibration board, and the image data is obtained by an image acquisition device acquisition calibration board.

[0206] The first point cloud data determination module 62 is used to process the point cloud data for each frame of point cloud data to obtain the point cloud data corresponding to the calibration board, and to translate and rotate the point cloud data corresponding to the calibration board to a preset plane to obtain the corresponding first point cloud data.

[0207] The corner coordinate determination module 63 is used to cluster each of the first point cloud data to obtain at least one cluster, determine the contour point coordinates of each cluster based on the second point cloud data in each cluster, determine the contour point list based on the contour point coordinates, and determine the point cloud corner coordinates of the board based on the contour point list.

[0208] The conversion relationship determination module 64 is used to determine the corner coordinates of the image based on each of the image data, and to determine the conversion relationship between the radar and the image acquisition device based on the corner coordinates of the point cloud and the corner coordinates of the image.

[0209] This invention provides a radar and camera calibration device that solves the problem of inaccurate joint calibration of camera and radar. After acquiring point cloud data and its corresponding image data, the point cloud data is first processed to filter out invalid point clouds and retain valid point clouds to obtain the point cloud data corresponding to the calibration board. The point cloud data corresponding to the calibration board is then moved to a preset plane through rotation and translation operations to obtain the first point cloud data. The first point cloud data is clustered to obtain clusters. The contour point coordinates of each cluster are determined based on the second point cloud data in each cluster. A contour point list is formed based on the contour point coordinates, and then the corner point coordinates of the calibration board are determined based on the contour point list. The coordinates of the corner points of the calibration board are automatically determined, and the results are accurate. The corner point coordinates of the image are determined through image data. Finally, the conversion relationship between the radar and the image acquisition device is determined based on the corner point coordinates of each point cloud and the corner point coordinates of the image, thus completing the calibration of the radar and camera. The calibration method provided in this application is simple in process and accurate in result.

[0210] Optionally, the data acquisition module 61 includes:

[0211] The timestamp determination unit is used to determine the timestamp corresponding to each frame of point cloud data.

[0212] The image data determination unit is used to filter out the image data corresponding to the point cloud data based on the timestamp.

[0213] Optionally, the first point cloud data determination module 62 includes:

[0214] The candidate point cloud determination unit is used to filter the point cloud data according to the point cloud range corresponding to the calibration board and determine the candidate point cloud.

[0215] The target point cloud determination unit is used to filter the candidate point clouds and determine the target point cloud;

[0216] The plane fitting unit is used to perform plane fitting based on the target point cloud, determine the plane equation of the fitting plane, and filter out the point cloud in the plane based on the plane equation.

[0217] The projection unit is used to project the point cloud in the plane onto the fitting plane to obtain the point cloud data corresponding to the calibration plate.

[0218] Optionally, the first point cloud data determination module 62 includes:

[0219] The translation unit is used to filter out reference points from the point cloud data corresponding to the calibration board, and subtract the coordinates of the reference points from the point cloud data corresponding to the calibration board to obtain the translated point cloud data.

[0220] A rotation matrix determination unit is used to determine the first rotation matrix and the second rotation matrix based on the plane equation of the fitted plane;

[0221] A rotation unit is used to rotate the translated point cloud data to a preset plane based on the first rotation matrix and the second rotation matrix to obtain the corresponding first point cloud data.

[0222] Optionally, the rotation matrix determining unit is specifically used to determine the plane normal vector according to the plane equation of the fitted plane, determine the first rotation parameter according to the plane normal vector, and determine the first rotation matrix based on the first rotation parameter; determine the normal vector of the rotated plane according to the first rotation parameter, determine the second rotation parameter according to the normal vector of the rotated plane, and determine the second rotation matrix based on the second rotation parameter.

[0223] Optionally, the corner coordinate determination module 63 includes:

[0224] A center point determination unit is used to determine the center point for each cluster based on the second point cloud data in the cluster.

[0225] The first contour point determination unit is used to calculate the first distance between each second point cloud data and the center point, and to determine the second point cloud data corresponding to the farthest first distance as the first contour point.

[0226] The second contour point determination unit is used to calculate the second distance between each second point cloud data and the first contour point, and to determine the second point cloud data corresponding to the second distance with the farthest second distance as the second contour point.

[0227] The contour point coordinate determination unit is used to determine the contour point coordinates of the cluster based on the coordinates of the first contour point and the coordinates of the second contour point.

[0228] Optionally, the corner coordinate determination module 63 includes:

[0229] The straight line direction determination unit is used to calculate the distance between the contour point in the next cluster and the starting point, taking a contour point in the first cluster as the starting point, taking the contour point with the closest distance as the target contour point, and determining the straight line direction between the target contour point and the starting point.

[0230] The first list forming unit is used to generate a list of contour points based on the contour point corresponding to the starting point and the target contour points with the same straight line direction if the straight line direction changes and the number of all target contour points meets the requirements, take the latest target contour point as the new starting point, return to perform the steps of calculating the distance between the contour point in the next cluster and the starting point, take the contour point with the closest distance as the target contour point, and determine the straight line direction between the target contour point and the starting point.

[0231] The contour point filtering unit is used to filter out the target contour points if the straight line direction changes and the number of target contour points does not meet the requirements, take the latest unfiltered target contour point as the new starting point, return to calculate the distance between the contour points in the next cluster and the starting point, take the closest contour point as the target contour point, and determine the straight line direction between the target contour point and the starting point.

[0232] The starting point update unit is used to determine the straight line direction between the target contour point and the starting point if the straight line direction has not changed and the target contour point is not the contour point in the last cluster. The latest target contour point is used as the new starting point, and the unit returns to calculate the distance between the contour point in the next cluster and the starting point, and takes the nearest contour point as the target contour point.

[0233] The second list forming unit is used to generate a list of contour points based on the contour point corresponding to the starting point and each target contour point if the straight line direction has not changed and the target contour point is a contour point in the last cluster. It determines whether all contour points have been traversed. If so, it confirms that all contour point lists have been generated. Otherwise, it takes another contour point in the first cluster as the new starting point, returns to calculate the distance between the contour point in the next cluster and the starting point, takes the closest contour point as the target contour point, and determines the straight line direction between the target contour point and the starting point.

[0234] Optionally, the corner coordinate determination module 63 includes:

[0235] The line fitting unit is used to perform line fitting on the contour points in each of the contour point lists to determine the line equation of the contour line.

[0236] The intersection point calculation unit is used to calculate the intersection point based on the straight line equations of each of the contour points, and obtain the coordinates of the corner point.

[0237] The inverse transformation unit is used to perform an inverse transformation on the coordinates of each corner point based on the first rotation matrix and the second rotation matrix to obtain the transformed corner point coordinates.

[0238] The corner coordinate determination unit is used to add the reference point coordinates to the transformed corner coordinates to obtain the corner coordinates of the point cloud.

[0239] The radar and camera calibration device provided in the embodiments of the present invention can execute the radar and camera calibration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0240] Example 6

[0241] Figure 7 This is a schematic diagram of a radar-camera coordinate transformation device provided in Embodiment Six of the present invention. Figure 7 As shown, the device includes: a data acquisition module 71 to be converted and a coordinate transformation module 72;

[0242] Among them, the data acquisition module 71 is used to acquire point cloud data to be converted, which is collected by radar;

[0243] The coordinate transformation module 72 is used to perform coordinate transformation on the point cloud data to be transformed according to the transformation relationship between the radar and the image acquisition device to obtain target data. The transformation relationship between the radar and the image acquisition device is determined by the radar and camera coordinate calibration method according to any embodiment of the present invention.

[0244] The radar and camera coordinate transformation method of this invention can transform the coordinates of point cloud data acquired by radar. By using the method provided in any embodiment of this application to determine the transformation relationship between radar and image acquisition device, the coordinate transformation of point cloud data is performed. Since the calibration result is accurate, accurate target data can be obtained when performing coordinate transformation. It can be applied in scenarios with high data accuracy, and the application scenarios are more extensive. When applied in the field of intelligent driving, it provides higher safety and a better user experience.

[0245] The radar-camera coordinate transformation device provided in the embodiments of the present invention can execute the radar-camera coordinate transformation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0246] Example 7

[0247] Figure 8This is a schematic diagram of the structure of an electronic device 80 provided in Embodiment 7 of the present invention. This electronic device can be used to implement the methods provided in the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0248] like Figure 8 As shown, the electronic device 80 includes at least one processor 81 and a memory, such as a read-only memory (ROM) 82 and a random access memory (RAM) 83, communicatively connected to the at least one processor 81. The memory stores computer programs executable by the at least one processor. The processor 81 can perform various appropriate actions and processes based on the computer program stored in the ROM 82 or loaded from storage unit 88 into the RAM 83. The RAM 83 can also store various programs and data required for the operation of the electronic device 80. The processor 81, ROM 82, and RAM 83 are interconnected via a bus 84. An input / output (I / O) interface 85 is also connected to the bus 84.

[0249] Multiple components in electronic device 80 are connected to I / O interface 85, including: input unit 86, such as keyboard, mouse, etc.; output unit 87, such as various types of monitors, speakers, etc.; storage unit 88, such as disk, optical disk, etc.; and communication unit 89, such as network card, modem, wireless transceiver, etc. Communication unit 89 allows electronic device 80 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0250] Processor 81 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 81 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 81 performs the various methods and processes described above, such as radar and camera calibration methods or radar and camera coordinate transformation methods.

[0251] In some embodiments, the radar and camera calibration method or the radar and camera coordinate transformation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 88. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 80 via ROM 82 and / or communication unit 89. When the computer program is loaded into RAM 83 and executed by processor 81, one or more steps of the radar and camera calibration method or the radar and camera coordinate transformation method described above may be performed. Alternatively, in other embodiments, processor 81 may be configured to perform the radar and camera calibration method or the radar and camera coordinate transformation method by any other suitable means (e.g., by means of firmware).

[0252] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0253] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0254] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0255] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0256] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0257] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0258] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0259] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A calibration method for radar and camera, characterized in that, include: Acquire at least one frame of point cloud data and image data corresponding to each of the point cloud data, wherein the point cloud data is obtained by a radar acquisition calibration board and the image data is obtained by an image acquisition device acquisition calibration board; For each frame of point cloud data, the point cloud data is processed to obtain the point cloud data corresponding to the calibration board. The point cloud data corresponding to the calibration board is translated and rotated to a preset plane to obtain the corresponding first point cloud data. Cluster the first point cloud data to obtain at least one cluster, determine the contour point coordinates of each cluster based on the second point cloud data in each cluster, determine the contour point list based on the contour point coordinates, and determine the point cloud corner point coordinates of the calibration board based on the contour point list. The corner coordinates of the images are determined based on the image data, and the conversion relationship between the radar and the image acquisition device is determined based on the corner coordinates of the point cloud and the image corner coordinates. The step of determining the contour point list based on the coordinates of each contour point includes: Starting from a contour point in the first cluster, calculate the distance between the contour point in the next cluster and the starting point, take the contour point with the closest distance as the target contour point, and determine the straight line direction between the target contour point and the starting point. If the direction of the straight line changes and the number of all target contour points meets the requirements, a list of contour points is generated based on the contour point corresponding to the starting point and each target contour point with the same straight line direction. The latest target contour point is taken as the new starting point. The process is to return to calculate the distance between the contour point in the next cluster and the starting point, take the contour point with the closest distance as the target contour point, and determine the straight line direction between the target contour point and the starting point. If the direction of the straight line changes and the number of target contour points does not meet the requirements, filter out the target contour points, take the latest unfiltered target contour point as the new starting point, return to execute the steps of calculating the distance between the contour point in the next cluster and the starting point, take the contour point with the closest distance as the target contour point, and determine the direction of the straight line between the target contour point and the starting point. If the straight line direction has not changed and the target contour point is not a contour point in the last cluster, the latest target contour point is taken as the new starting point, and the process returns to calculate the distance between the contour point in the next cluster and the starting point, taking the contour point closest to the target contour point as the target contour point, and determining the straight line direction between the target contour point and the starting point. If the straight line direction has not changed and the target contour point is the contour point in the last cluster, generate a contour point list based on the contour point corresponding to the starting point and each target contour point. Determine whether to traverse all contour points. If yes, confirm that all contour point lists have been generated. Otherwise, take another contour point in the first cluster as the new starting point, return to calculate the distance between the contour point in the next cluster and the starting point, take the closest contour point as the target contour point, and determine the straight line direction between the target contour point and the starting point.

2. The method according to claim 1, characterized in that, Obtaining the image data corresponding to each of the point cloud data includes: For each frame of point cloud data, determine the timestamp corresponding to the point cloud data; Image data corresponding to the point cloud data is filtered out based on the timestamp.

3. The method according to claim 1, characterized in that, The process of processing the point cloud data to obtain the point cloud data corresponding to the calibration board includes: The point cloud data is filtered according to the point cloud range corresponding to the calibration board to determine candidate point clouds; The candidate point clouds are filtered to determine the target point cloud; Based on the target point cloud, a plane fit is performed to determine the plane equation of the fitted plane, and the point cloud in the plane is selected based on the plane equation. The point cloud in the plane is projected onto the fitting plane to obtain the point cloud data corresponding to the calibration board.

4. The method according to claim 3, characterized in that, The step of translating and rotating the point cloud data corresponding to the calibration board to a preset plane to obtain the corresponding first point cloud data includes: The reference point is selected from the point cloud data corresponding to the calibration board, and the coordinates of the reference point are subtracted from the point cloud data corresponding to the calibration board to obtain the translated point cloud data. The first rotation matrix and the second rotation matrix are determined based on the plane equation of the fitted plane; Based on the first rotation matrix and the second rotation matrix, the translated point cloud data is rotated to a preset plane to obtain the corresponding first point cloud data.

5. The method according to claim 4, characterized in that, The step of determining the first rotation matrix and the second rotation matrix based on the plane equation of the fitted plane includes: The plane normal vector is determined based on the plane equation of the fitted plane, the first rotation parameter is determined based on the plane normal vector, and the first rotation matrix is ​​determined based on the first rotation parameter. The normal vector of the rotated plane is determined based on the first rotation parameter, the second rotation parameter is determined based on the normal vector of the rotated plane, and the second rotation matrix is ​​determined based on the second rotation parameter.

6. The method according to claim 1, characterized in that, Determining the contour point coordinates of each cluster based on the second point cloud data in each cluster includes: For each cluster, the center point is determined based on the second point cloud data in the cluster; Calculate the first distance between each second point cloud data and the center point, and determine the second point cloud data corresponding to the farthest first distance as the first contour point; Calculate the second distance between each second point cloud data and the first contour point, and determine the second point cloud data corresponding to the second distance with the farthest second distance as the second contour point; The coordinates of the contour points of the cluster are determined based on the coordinates of the first contour point and the coordinates of the second contour point.

7. The method according to claim 4, characterized in that, The step of determining the point cloud corner coordinates of the calibration board based on the list of contour points includes: Perform line fitting on the contour points in each of the contour point lists to determine the straight line equation of the contour line. Calculate the intersection points based on the linear equations of each contour point to obtain the corner coordinates; Based on the first rotation matrix and the second rotation matrix, the coordinates of each corner point are inversely transformed to obtain the transformed corner point coordinates. Adding the coordinates of the reference point to the transformed corner points yields the corner point coordinates of the point cloud.

8. A coordinate transformation method between radar and camera, characterized in that, include: Acquire point cloud data to be converted, which is collected by radar; The point cloud data to be converted is subjected to coordinate transformation according to the conversion relationship between the radar and the image acquisition device to obtain target data, wherein the conversion relationship between the radar and the image acquisition device is determined according to the radar and camera calibration method according to any one of claims 1-7.

9. A calibration device for radar and camera, characterized in that, include: The data acquisition module is used to acquire at least one frame of point cloud data and image data corresponding to each of the point cloud data. The point cloud data is obtained by a radar acquisition calibration board, and the image data is obtained by an image acquisition device acquisition calibration board. The first point cloud data determination module is used to process the point cloud data for each frame of point cloud data to obtain the point cloud data corresponding to the calibration board, and to translate and rotate the point cloud data corresponding to the calibration board to a preset plane to obtain the corresponding first point cloud data. The corner coordinate determination module is used to cluster each of the first point cloud data to obtain at least one cluster, determine the contour point coordinates of each cluster based on the second point cloud data in each cluster, determine the contour point list based on the contour point coordinates, and determine the point cloud corner coordinates of the calibration board based on the contour point list. The conversion relationship determination module is used to determine the corner coordinates of the image based on each of the image data, and to determine the conversion relationship between the radar and the image acquisition device based on the corner coordinates of the point cloud and the corner coordinates of the image. The corner coordinate determination module further includes: The straight line direction determination unit is used to calculate the distance between the contour point in the next cluster and the starting point, taking a contour point in the first cluster as the starting point, taking the contour point with the closest distance as the target contour point, and determining the straight line direction between the target contour point and the starting point. The first list forming unit is used to generate a list of contour points based on the contour point corresponding to the starting point and the target contour points with the same straight line direction if the straight line direction changes and the number of all target contour points meets the requirements, take the latest target contour point as the new starting point, return to perform the steps of calculating the distance between the contour point in the next cluster and the starting point, take the contour point with the closest distance as the target contour point, and determine the straight line direction between the target contour point and the starting point. The contour point filtering unit is used to filter out the target contour points if the straight line direction changes and the number of target contour points does not meet the requirements, take the latest unfiltered target contour point as the new starting point, return to calculate the distance between the contour points in the next cluster and the starting point, take the closest contour point as the target contour point, and determine the straight line direction between the target contour point and the starting point. The starting point update unit is used to, if the straight line direction has not changed and the target contour point is not a contour point in the last cluster, take the latest target contour point as the new starting point, return to calculate the distance between the contour point in the next cluster and the starting point, take the nearest contour point as the target contour point, and determine the straight line direction between the target contour point and the starting point. The second list forming unit is used to generate a list of contour points based on the contour point corresponding to the starting point and each target contour point if the straight line direction has not changed and the target contour point is a contour point in the last cluster. It determines whether all contour points have been traversed. If so, it confirms that all contour point lists have been generated. Otherwise, it takes another contour point in the first cluster as the new starting point, returns to calculate the distance between the contour point in the next cluster and the starting point, takes the closest contour point as the target contour point, and determines the straight line direction between the target contour point and the starting point.

10. A coordinate transformation device for radar and camera, characterized in that, include: The data to be converted acquisition module is used to acquire point cloud data to be converted, which is collected by radar. A coordinate transformation module is used to perform coordinate transformation on the point cloud data to be transformed according to the transformation relationship between the radar and the image acquisition device to obtain target data, wherein the transformation relationship between the radar and the image acquisition device is determined according to the radar and camera calibration method according to any one of claims 1-7.

11. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method of any one of claims 1-8.

Images