Radar and camera joint calibration method, device, equipment and medium

Abstract

The invention relates to a radar and camera joint calibration method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring a three-dimensional point cloud of each plane on a target through a radar, and acquiring image data of the target through a camera; the target is formed by splicing at least three planes at an included angle of non-180 degrees, and splicing lines are formed among the spliced planes; performing plane fitting on the three-dimensional point cloud to obtain a fitting plane used for representing each plane of the target under a radar point cloud coordinate system, and determining first information based on the fitting plane, the first information being used for representing the position and / or attitude of the calibration reference object under the radar point cloud coordinate system; identifying second information in the image data; and according to the first information and the second information, determining a conversion relation between a radar point cloud coordinate system and a camera coordinate system. Therefore, the calibration precision between the radar point cloud coordinate system and the camera coordinate system can be effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of radar data processing, and in particular to a radar and camera joint calibration method, apparatus, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] With the development of radar technology, more and more vehicles are equipped with lidar and cameras. However, since the coordinate systems corresponding to the data of the two are different, the data of the two need to be calibrated when used.

[0003] In traditional technology, there are two main calibration methods: Method 1: Use laser radar and camera to detect the target grid respectively, so as to locate the coordinates of multiple grid vertices, and then calculate the corresponding conversion relationship based on the coordinates of the grid vertices in the two coordinate systems. Method 2: Use a round hole board as a calibration board, judge the position of the hole on the calibration board by the presence or absence of points, and then calculate the center of the circle.

[0004] However, for method 1, the accuracy of laser radar in identifying mesh vertices is often poor, which results in the coordinates of the located mesh vertices being inaccurate or inaccurate. On this basis, the calculated conversion relationship is also inaccurate. For method 2, although the center position of the circle can be obtained by fitting a circle, the number of circular holes that can exist on a calibration plate is limited, which will result in limited input parameters for obtaining the external parameters of the laser radar and camera. In addition, since the points of the laser point cloud are discrete, when fitting the center of the circle, the fitting accuracy will be low and it is easy to affect the accuracy of the external parameters. Summary of the invention

[0005] Based on this, it is necessary to provide a radar and camera joint calibration method, device, computer equipment, computer-readable storage medium and computer program product that can effectively improve the calibration accuracy in response to the above technical problems.

[0006] In a first aspect, the present application provides a joint calibration method of a radar and a camera, comprising:

[0007] Acquire three-dimensional point clouds of each plane on the target through a radar, and acquire image data of the target through a camera; wherein the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes, and the splicing line and / or points on the splicing line are used as calibration reference objects;

[0008] Performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system, and determining first information based on the fitting plane, where the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system;

[0009] identifying second information in the image data, the second information being used to characterize a position and / or a posture of the calibration reference object in a camera coordinate system;

[0010] A conversion relationship between a radar point cloud coordinate system and a camera coordinate system is determined according to the first information and the second information.

[0011] In one embodiment, the calibration reference object includes at least one of the following:

[0012] A designated intersection point formed by the common intersection of three designated planes in the target;

[0013] The calibration reference object includes a designated stitching line formed by two designated planes in the target;

[0014] The endpoints of the designated splicing line;

[0015] The designated points for dividing the stitching line equally;

[0016] If the calibration reference object includes the designated intersection point, the first information includes the coordinates of the fitting intersection point used to characterize the designated intersection point in the radar point cloud coordinate system;

[0017] If the calibration reference object includes the designated stitching line, the first information includes a line equation for representing a fitting intersection line of the designated stitching line in the radar point cloud coordinate system;

[0018] If the calibration reference object includes the endpoints of the designated stitching line, the first information includes the coordinates of the endpoints of the fitted intersection line representing the designated stitching line in the radar point cloud coordinate system;

[0019] If the calibration reference object includes equally divided points of the designated stitching line, the first information includes coordinates of equally divided points of the fitted intersection line representing the designated stitching line in the radar point cloud coordinate system.

[0020] In one embodiment, after performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in the radar point cloud coordinate system, the method further includes:

[0021] According to the fitting structural parameters associated with the fitting intersection points and / or the fitting intersection lines, and the actual structural parameters of the target known in advance, the fitting accuracy is evaluated to obtain an evaluation result; the conversion relationship is determined when the evaluation result meets the preset standard. In one embodiment, the actual structural parameters include a first distance between designated intersection points in the target; the designated intersection points are formed by the intersection of three designated planes in the target; the fitting structural parameters include a second distance between the fitting intersection points used to characterize the designated intersection points in the radar point cloud coordinate system; each of the fitting intersection points is formed by the intersection of three fitting planes; the evaluation result includes a distance error between the first distance and the second distance;

[0022] and / or:

[0023] The actual structural parameters include a first angle between two designated planes spliced ​​together in the target, the fitting structural parameters include a second angle between the fitting planes spliced ​​together, and the evaluation result includes an angle error between the first angle and the second angle.

[0024] In one embodiment, the method further includes: if the distance error is greater than a first preset threshold, adjusting the target's posture and then re-collecting the three-dimensional point cloud through the radar, and performing plane fitting based on the re-collected three-dimensional point cloud until the distance error between each plane intersection is no greater than the first preset threshold; and / or,

[0025] If the angle error is greater than a second preset threshold, the target's position and posture are adjusted and the three-dimensional point cloud is collected again by radar, and plane fitting is performed based on the re-collected three-dimensional point cloud until the angle error between the planes is no greater than the second preset threshold, so as to obtain the fitting planes of the target's planes.

[0026] In one embodiment, the at least three planes in the target include a first plane and multiple second planes, the first plane is polygonal, each of the second planes is spliced ​​to a straight edge of the first plane, and the second planes formed by two adjacent straight edges of the first plane are also spliced ​​to each other, and the angle between the first plane and the second plane is an obtuse angle.

[0027] In one embodiment, performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system includes:

[0028] Determining at least three manually selected points in the three-dimensional point cloud, wherein the at least three points are determined as points used to represent the first plane in the radar point cloud coordinate system;

[0029] Based on the at least three points, obtain a first fitting plane for representing the first plane by fitting;

[0030] Determine a bounding box of the point corresponding to the first fitting plane, wherein the shape of the bounding box is the same as that of the first plane, and each side of the bounding box corresponds to an side of the first plane;

[0031] A group of point clouds not belonging to the first fitting plane is selected within a specified range near each side of the bounding box, and a second fitting plane for representing a second plane is fitted based on each selected group of point clouds.

[0032] In one embodiment, performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system includes:

[0033] Randomly select a number of points in the target point cloud in the three-dimensional point cloud, and then perform fitting based on the number of points and the point cloud in the target point cloud;

[0034] If the plane cannot be fitted, select several new points;

[0035] If a plane can be fitted, the fitted plane is determined to be the fitting plane, points corresponding to the fitting plane are screened out from the target point cloud, and the remaining point cloud is used as a new target point cloud;

[0036] The above process is repeated to obtain a fitting plane for characterizing each plane of the target.

[0037] In one embodiment, performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system includes any of the following methods:

[0038] Method 1: clustering the three-dimensional point cloud to obtain multiple clusters, wherein the three-dimensional point cloud in each cluster corresponds to a plane;

[0039] Selecting a cluster as a target cluster, performing plane fitting on the three-dimensional point cloud in the target cluster to obtain a first fitting plane;

[0040] Determine the distance between each point in the remaining three-dimensional point cloud of other clusters and the first fitting plane respectively, and screen the three-dimensional point clouds in the other clusters according to the distance to obtain the target point cloud; perform plane fitting according to the target point cloud to obtain the second fitting plane corresponding to the other clusters; wherein, the farther the distance between the three-dimensional point cloud in the other cluster and the first fitting plane is, the higher the probability of being selected as the target point cloud;

[0041] Method 2: clustering the three-dimensional point cloud to obtain multiple clusters, wherein the three-dimensional point cloud in each cluster corresponds to a plane;

[0042] Selecting a cluster as a target cluster, performing plane fitting on the three-dimensional point cloud in the target cluster to obtain a first fitting plane;

[0043] Determine the distance between each point in the three-dimensional point cloud of the remaining other clusters and the first fitting plane, and divide the other clusters into at least two point cloud groups according to the distance; perform plane fitting on the three-dimensional point cloud in each point cloud group to obtain the plane equation of the sub-fitting plane corresponding to each point cloud group; perform weighted summation on the coefficients of the plane equation of the sub-fitting plane corresponding to each point cloud group to obtain the plane equation of the second fitting plane corresponding to the other clusters, wherein the sub-fitting plane corresponding to the point cloud group that is closer to the first fitting plane has a lower weight.

[0044] In one embodiment, the at least three planes in the target include a first plane and a plurality of second planes, the first plane is polygonal, each of the second planes is spliced ​​to a straight edge of the first plane, the second planes formed by two adjacent straight edges of the first plane are also spliced ​​to each other, and the angle between the first plane and the second plane is an obtuse angle; wherein the first plane intersects with any two second planes to form a designated intersection, and the calibration reference object includes at least three of the designated intersections;

[0045] The three-dimensional point cloud of the target cluster corresponds to the first plane, and the three-dimensional point clouds of the other clusters correspond to the second plane;

[0046] Based on the fitting plane, determining first information includes:

[0047] Based on the fitting intersection line between the second fitting planes and the pre-known length of the stitching line between the second planes in the target, the coordinates of the fitting intersection point used to characterize the designated intersection point in the radar point cloud coordinate system are determined as part or all of the first information.

[0048] In a second aspect, the present application also provides a radar and camera joint calibration device, including:

[0049] An acquisition module, used to obtain three-dimensional point clouds of each plane on the target through a radar, and to obtain image data of the target through a camera; the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes, and the splicing line and / or points on the splicing line are used as calibration reference objects;

[0050] a fitting module, configured to perform plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system, and determine first information based on the fitting plane, wherein the first information is used to characterize a position and / or posture of the calibration reference object in the radar point cloud coordinate system;

[0051] a positioning module, configured to identify second information in the image data, wherein the second information is used to characterize the position and / or posture of the calibration reference object in a camera coordinate system;

[0052] A conversion module is used to determine a conversion relationship between a radar point cloud coordinate system and a camera coordinate system according to the first information and the second information.

[0053] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0054] Acquire three-dimensional point clouds of each plane on the target through a radar, and acquire image data of the target through a camera; wherein the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes, and the splicing line and / or points on the splicing line are used as calibration reference objects;

[0055] Performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system, and determining first information based on the fitting plane, where the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system;

[0056] identifying second information in the image data, the second information being used to characterize a position and / or a posture of the calibration reference object in a camera coordinate system;

[0057] According to the first information and the second information, the conversion relationship between the radar point cloud coordinate system and the camera coordinate system is determined. In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0058] Acquire three-dimensional point clouds of each plane on the target through a radar, and acquire image data of the target through a camera; wherein the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes, and the splicing line and / or points on the splicing line are used as calibration reference objects;

[0059] Performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system, and determining first information based on the fitting plane, where the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system;

[0060] identifying second information in the image data, the second information being used to characterize a position and / or a posture of the calibration reference object in a camera coordinate system;

[0061] A conversion relationship between a radar point cloud coordinate system and a camera coordinate system is determined according to the first information and the second information.

[0062] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0063] Acquire three-dimensional point clouds of each plane on the target through a radar, and acquire image data of the target through a camera; wherein the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes, and the splicing line and / or points on the splicing line are used as calibration reference objects;

[0064] Performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system, and determining first information based on the fitting plane, where the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system;

[0065] identifying second information in the image data, the second information being used to characterize a position and / or a posture of the calibration reference object in a camera coordinate system;

[0066] A conversion relationship between a radar point cloud coordinate system and a camera coordinate system is determined according to the first information and the second information.

[0067] The above-mentioned radar and camera joint calibration method, device, computer equipment, computer-readable storage medium and computer program product obtain the three-dimensional point cloud of each plane on the target through the radar, and obtain the image data of the target through the camera; wherein, the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and the spliced ​​planes form a splicing line; the splicing line and / or the points on the splicing line are used as calibration reference objects; thus, the three-dimensional point cloud can be obtained by the radar, and compared with the two-dimensional calibration object, the detected point cloud is obtained based on the distance, and there is no need to infer based on parameters such as reflectivity, so the accuracy of the subsequent conversion relationship can be improved. The three-dimensional point cloud is plane-fitted to obtain a fitting plane for characterizing each plane of the target in the radar point cloud coordinate system, and based on the fitting plane, the first information is determined, and the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system; thus, the plane equation of the target can be obtained according to the three-dimensional point cloud fitting, and the intersection, end point, line equation, etc. of the splicing line are calculated by the plane equation, which is convenient for the subsequent calculation of the conversion relationship between different coordinate systems. Identify the second information in the image data, the second information is used to characterize the position and / or posture of the calibration reference object in the camera coordinate system; thereby, the intersection points, endpoints, line equations, etc. corresponding to the aforementioned three-dimensional point cloud can be obtained, which are used to achieve accurate point / line conversion. According to the first information and the second information, the conversion relationship between the radar point cloud coordinate system and the camera coordinate system is determined. Thus, the position of each plane of the target can be accurately reflected by the three-dimensional point cloud collected by the radar. Since the plane on which the conversion relationship depends is fitted, there is no need to use discrete point clouds to distinguish the dividing line, thereby effectively improving the calibration accuracy between the radar point cloud coordinate system and the camera coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 A schematic diagram of the structure of a target used in a joint calibration method of a radar and a camera in one embodiment;

[0070] Figure 2 It is a schematic diagram of the structure of a target in an actual product in an embodiment;

[0071] Figure 3 is a schematic flow chart of a joint calibration method for a radar and a camera in one embodiment;

[0072] Figure 4 is a schematic flow chart of a joint calibration method for radar and camera in another embodiment;

[0073] Figure 5 is a schematic flow chart of a joint calibration method of a radar and a camera in yet another embodiment;

[0074] Figure 6 is a structural block diagram of a joint calibration device for radar and camera in one embodiment;

[0075] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0077] The combined calibration method of radar and camera provided in the embodiment of the present application can be used as follows Figure 1 The targets shown are used for joint calibration. Figure 1 As shown, the target comprises four triangle plates A, B, C, and D, wherein the triangle plates A, B, and C are spliced ​​into a triangular pyramid shape, and the triangle plate D is embedded in the triangular pyramid and spliced ​​with the triangle plates A, B, and C. After the four triangle plates are spliced, six splicing lines L1, L2, L3, L4, L5, and L6 can be formed, and the target can be fixedly connected or hinged to the outside through a fixing piece.

[0078] In an exemplary embodiment, first, after fixing the position of the target, trigger the camera and the laser radar to detect the target from the opening of the target. For the camera, after the camera shoots, the lines of L1, L2, L3, L4, L5, and L6 can be clearly reflected in the captured image. Optionally, for the convenience of reflection, the materials and colors of the four triangles A, B, C, and D can be set to be differentiated as much as possible, so as to more clearly reflect the lines of L1, L2, L3, L4, L5, and L6, so as to locate L1, L2, L3, L4, L5, and L6, or the intersections and endpoints between L1, L2, L3, L4, L5, and L6 in the image acquired by the camera, and then obtain the coordinates of these points or line expressions in the coordinate system of the camera. For the laser radar, when the laser radar scans a three-dimensional object such as a target, the point cloud of the three-dimensional object can be obtained based on the distance of each point on the three-dimensional object relative to the laser radar. Exemplary, in detection Figure 1When the target is marked as shown, the point clouds of the surfaces of the four plates A, B, C, and D can be obtained. Through the plane fitting of the laser point cloud, the mathematical expressions of the four planes A, B, C, and D can be obtained. Based on these planes, the expressions of L1, L2, L3, L4, L5, and L6 can be located in the radar point cloud coordinate system, or: the coordinates of the intersections and endpoints between L1, L2, L3, L4, L5, and L6. Finally, based on the points and / or lines identified in the radar point cloud coordinate system and the camera coordinate system, the conversion relationship between the two coordinate systems is calculated.

[0079] It should be noted that Figure 1 The shape of the target shown satisfies the constraints of the extrinsic degrees of freedom to the greatest extent possible. In comparison, a cube or origami-shaped target would have insufficient constraints on one or more degrees of freedom. For example, Figure 1 In the target shape shown, the six splicing lines L1, L2, L3, L4, L5, and L6 are in different postures and are not parallel to each other, which provides more constraints on the degrees of freedom and thus can ensure the accuracy of the desired conversion relationship.

[0080] In an exemplary embodiment, Figure 2 As shown in the figure, it shows the schematic diagram of the structure of the target in the actual product. The color changes on different sides of the target are to better identify points and lines in the image; the QR code in the center of the target is used to determine its position and direction in the camera image.

[0081] For example, Figure 1 As shown, the target can be composed of a plurality of triangular plates, wherein the triangular plates can be made of Chevron plate material, and the target bracket can be composed of three aluminum profiles with a side length of about 1 meter. Figure 1 Taking the target in as an example, an aluminum profile is set at the junction of plate A and plate C, an aluminum profile is set at the junction of plate A and plate B, and an aluminum profile is set at the junction of plate B and plate C, and the triangular plate is connected to the aluminum profile; the three aluminum profiles are connected to the outside through fixings, and the aluminum profiles and the fixings can be hinged, which makes it easy to adjust the posture of the entire target, thereby ensuring that the camera and laser radar can detect the bell mouth of the target.

[0082] It should be noted that this embodiment does not limit the shape of the planes spliced ​​into the target (for example, various polygonal corner plates) and the splicing form between the planes. As long as there are several planes with splicing lines, the laser radar can fit the corresponding planes. Therefore, the shape of the target must meet the following requirements: it contains at least three planes, and the spliced ​​planes in the at least three planes can form a splicing line, thereby obtaining three or more splicing lines (for example, the target can be in the shape of a triangular pyramid).

[0083] In an exemplary embodiment, Figure 3As shown, a joint calibration method of radar and camera is provided, and the method is applied to Figure 1 The target in is taken as an example to illustrate, including the following steps 301 to 304. Among them:

[0084] Step 301, obtaining three-dimensional point clouds of each plane on the target through a radar, and obtaining image data of the target through a camera.

[0085] The target is formed by splicing at least three planes at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes. The splicing line and / or points on the splicing line are used as calibration reference objects.

[0086] In an optional embodiment, the at least three planes in the target include a first plane and multiple second planes, the first plane is polygonal, each of the second planes is spliced ​​to a straight edge of the first plane, the second planes formed by two adjacent straight edges of the first plane are also spliced ​​to each other, the angle between the first plane and the second plane is an obtuse angle, and the angle between the second planes is an acute angle.

[0087] Exemplarily, the stitching line and / or the point on the stitching line as the calibration reference object may include at least one of the following:

[0088] A designated intersection point formed by the common intersection of three designated planes in the target;

[0089] The calibration reference object includes a designated stitching line formed by two designated planes in the target;

[0090] The endpoints of the designated splicing line;

[0091] The designated points for dividing the stitching line equally;

[0092] If the calibration reference object includes the designated intersection point, the first information includes the coordinates of the fitting intersection point used to characterize the designated intersection point in the radar point cloud coordinate system;

[0093] If the calibration reference object includes the designated stitching line, the first information includes a line equation for representing a fitting intersection line of the designated stitching line in the radar point cloud coordinate system;

[0094] If the calibration reference object includes the endpoints of the designated stitching line, the first information includes the coordinates of the endpoints of the fitted intersection line representing the designated stitching line in the radar point cloud coordinate system;

[0095] If the calibration reference object includes equally divided points of the designated stitching line, the first information includes coordinates of equally divided points of the fitted intersection line representing the designated stitching line in the radar point cloud coordinate system.

[0096] See also Figure 1 Taking the target shown as an example, after the position of the target is fixed, the radar and the camera are triggered to collect the three-dimensional point clouds of each plane on the target and the image data of the target. Exemplarily, the radar and the camera can be triggered to detect from the opening of the target, and the three-dimensional point clouds of each plane on the target and the image data of the target are obtained respectively; wherein the image data of the target includes the stitching lines between the planes.

[0097] Step 302 , performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in the radar point cloud coordinate system, and determining first information based on the fitting plane.

[0098] In this embodiment, the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system. Figure 1 As shown, the at least three planes in the target include a first plane and multiple second planes, the first plane is polygonal, each second plane is spliced ​​to a straight edge of the first plane, and the second planes spliced ​​by two adjacent straight edges of the first plane are also spliced ​​to each other, and the angle between the first plane and the second plane is an obtuse angle; wherein the first plane intersects with any two second planes to form a designated intersection, and the calibration reference object includes at least three designated intersections; the three-dimensional point cloud of the target cluster corresponds to the first plane, and the three-dimensional point cloud of the other clusters corresponds to the second plane. wherein the angle between the second planes can be any one of an acute angle, a right angle, and an obtuse angle.

[0099] Wherein, based on the fitting plane, determining the first information includes: based on the fitting intersection line between the second fitting planes and the pre-known length of the stitching line between the second planes in the target, determining the coordinates of the fitting intersection point used to characterize the designated intersection point in the radar point cloud coordinate system as part or all of the first information.

[0100] In this embodiment, after the three-dimensional point clouds corresponding to each plane of the target are collected by the laser radar, plane fitting can be performed based on the three-dimensional point clouds to obtain the plane equations corresponding to each plane of the target.

[0101] In a first exemplary embodiment, at least three points manually selected can be determined in the three-dimensional point cloud, and the at least three points are determined as points used to represent the first plane in the radar point cloud coordinate system; based on the at least three points, a first fitting plane used to represent the first plane is fitted; a bounding box of the points corresponding to the first fitting plane is determined, the shape of the bounding box is the same as the first plane, and each edge of the bounding box corresponds to an edge of the first plane; a group of point clouds that do not belong to the first fitting plane are selected within a specified range near each edge of the bounding box, and based on each selected group of point clouds, a second fitting plane used to represent a second plane is fitted.

[0102] In a second exemplary embodiment, a number of points are randomly selected from the target point cloud in the three-dimensional point cloud, and then fitting is performed based on the number of points and the point cloud in the target point cloud; if a plane cannot be fitted, a number of points are re-selected; if a plane can be fitted, the fitted plane is determined to be the fitting plane, and the points corresponding to the fitting plane are screened out in the target point cloud, and the remaining point cloud is used as the new target point cloud; the above process is repeated to obtain the fitting plane used to characterize each plane of the target.

[0103] In a third exemplary embodiment, the three-dimensional point cloud is clustered to obtain a plurality of clusters, wherein the three-dimensional point cloud in each cluster corresponds to a plane. Then, a cluster is selected as a target cluster, and plane fitting is performed on the three-dimensional point cloud in the target cluster to obtain a first fitting plane; the distance between each point in the three-dimensional point cloud of the remaining other clusters and the first fitting plane is determined respectively, and the three-dimensional point clouds in the other clusters are screened according to the distance to obtain a target point cloud; plane fitting is performed on the target point cloud to obtain a second fitting plane corresponding to the other clusters; wherein, the farther the distance between the three-dimensional point cloud in the other cluster and the first fitting plane, the higher the probability of being selected as the target point cloud.

[0104] In this embodiment, before performing plane fitting, it is first necessary to divide the three-dimensional point cloud into lines, that is, to obtain the three-dimensional point clouds corresponding to each plane. For example, the three-dimensional point cloud can be clustered by clustering to obtain multiple clusters, each cluster corresponding to a plane. In addition, the three-dimensional point cloud can also be classified by the trained model, and each class corresponds to a plane. This embodiment does not limit the method of dividing the plane, and existing classification and clustering methods can be applied to this embodiment. In this embodiment, it can be combined with Figure 1The target shown in the figure is used as an example. Since the angles between planes (such as the angle between plane D and plane A, the angle between plane D and plane B, and the angle between plane D and plane C) are mostly obtuse angles, this will also cause the point cloud near the intersection of two planes to often be unable to accurately distinguish which plane it belongs to, resulting in the lack of accuracy of the fitted plane equation. To address this problem, after clustering the point cloud, at least four clusters can be obtained, and the point cloud in each cluster corresponds to a plane (with Figure 1 Taking the target shown as an example, the four clusters correspond to the four planes A, B, C, and D respectively). According to the relative position relationship of the point cloud, it can be determined which cluster is the point cloud of the D plane, which cluster is the point cloud of the A plane, which cluster is the point cloud of the B plane, and which cluster is the point cloud of the C plane. First, the point cloud of the D plane is plane-fitted to obtain the plane equation of the D plane. Then, the point cloud of the A plane can be screened. Specifically, according to the fitted D plane, the distance between each point in the point cloud of the A plane and the D plane can be calculated, and the points that are too close can be screened out. The remaining point clouds are used for fitting to obtain the plane equation of the A plane. Similarly, the point clouds of the B plane and the C plane can be processed in the same way to obtain the corresponding plane equations. In this process, for the A, B, and C planes, because the points that affect the accuracy are mainly the points near the boundary line, the nearby points are excluded, thereby effectively improving the fitting accuracy of the A, B, and C planes.

[0105] In a fourth exemplary embodiment, the three-dimensional point cloud can also be clustered to obtain multiple clusters, wherein the three-dimensional point cloud in each cluster corresponds to a plane. Then, a cluster is selected as a target cluster, and plane fitting is performed on the three-dimensional point cloud in the target cluster to obtain a first fitting plane; the distance between each point in the three-dimensional point cloud of the remaining other clusters and the first fitting plane is determined respectively, and the other clusters are divided into at least two point cloud groups according to the distance; plane fitting is performed on the three-dimensional point cloud in each point cloud group respectively to obtain the plane equation of the sub-fitting plane corresponding to each point cloud group; the coefficients of the plane equation of the sub-fitting plane corresponding to each point cloud group are weighted summed to obtain the plane equation of the second fitting plane corresponding to the other clusters, wherein the weighted weight of the sub-fitting plane corresponding to the point cloud group that is closer to the first fitting plane is lower.

[0106] In this embodiment, Figure 1The target shown in the figure is used as an example for explanation. First, the point cloud of the D plane is plane fitted to obtain the plane equation of the D plane. Then, the point cloud of the A plane can be divided into different point cloud groups according to the distance between each point in the point cloud of the A plane and the D plane. For example, if it is divided into two point cloud groups, one point cloud group is closer to the D plane and the other point cloud group is farther from the D plane. For different point cloud groups, plane fitting is performed separately to obtain multiple plane equations. Taking two point cloud groups as an example, there are two plane equations, as shown below:

[0107] K 1_1 *X+K 2_1 *Y+K 3_1 *Z+K 4_1 =0;

[0108] K 1_2 *X+K 2_2 *Y+K 3_2 *Z+K 4_2 =0;

[0109] Among them, X represents the X-axis variable, Y represents the Y-axis variable, and K 1_1 , K 2_1 , K 3_1 , K 4_1 , K 1_2 , K 2_2 , K 3_2 , K 4_2 Represents the coefficients of the plane equation.

[0110] Then, the coefficients of the plane equation of each point cloud group are weighted and summed to obtain the final plane equation. For example, for K 1_1 , K 1_2 Perform weighted summation and calculate K 2_1 , K 2_2 Perform weighted summation and calculate K 3_1 , K 3_2 A weighted sum is performed, wherein the closer to the D plane, the smaller the weighted value.

[0111] In a fifth exemplary embodiment, after the three-dimensional point cloud is divided into different planes, each plane may be fitted separately to obtain multiple plane equations.

[0112] In this embodiment, Figure 1 The target shown in the figure is used as an example for explanation. After fitting the four surfaces A, B, C, and D respectively, the four surface equations in the radar coordinate system can be obtained. The surface equations are three-variable linear equations.

[0113] Exemplarily, after obtaining the mathematical expressions of each plane, the mathematical expressions of each plane can be jointly solved to obtain the coordinates of the intersection points of each plane, or, in combination with the line equation, the coordinates of the endpoints and the equally divided points of each splicing line can be determined respectively according to the coordinates of the selected points, wherein the line equation is obtained according to the mathematical expressions of each plane.

[0114] In this embodiment, Figure 1 Take the target shown as an example for illustration. After solving the plane equations corresponding to the A, B, and D planes, the coordinates of the intersection of the A, B, and D planes (x1, y1) can be obtained; similarly, the coordinates of the intersection of the A, C, and D planes (x2, y2) and the intersection of the B, C, and D planes (x3, y3) can be obtained. Furthermore, the conversion relationship can be calculated based on the coordinates of the three intersections (x1, y1), (x2, y2), and (x3, y3) in the radar coordinate system and the coordinates of the corresponding intersections in the camera coordinate system.

[0115] Exemplarily, after the mathematical expressions of the planes are obtained, the corresponding line equations are obtained by combining any two three-variable linear equations in the mathematical expressions of the planes.

[0116] In this embodiment, the equation of the line can be combined with other points (known points) to calculate the coordinates of all endpoints, equally divided points, and other points of L1, L2, L3, L4, L5, and L6. Then, based on the coordinates of these points in the radar coordinate system and the coordinates of the corresponding points in the camera coordinate system, the conversion relationship can be calculated.

[0117] Step 303: Identify second information in the image data.

[0118] The second information is used to characterize the position and / or posture of the calibration reference object in the camera coordinate system.

[0119] In this embodiment, the camera has a camera coordinate system and an image coordinate system. The conversion relationship between the camera coordinate system and the image coordinate system is determined by the internal parameters of the camera, so it is generally known. When the conversion relationship between the camera coordinate system and the laser radar coordinate system (i.e., the coordinate system of the point cloud) needs to be calibrated, it still needs to be implemented in the image coordinate system. For example, when using the calibrated conversion relationship, the ultimate goal is to project the points in the laser radar coordinate system onto the image of the camera. Then, when determining the conversion relationship, if this conversion relationship is the conversion relationship between the camera coordinate system and the radar point cloud coordinate system, then after the calibration calculation, when using the conversion relationship, it is necessary to first use this conversion relationship to project the points to the camera coordinate system, and then use the conversion relationship between the camera coordinate system and the image coordinate system determined by the internal parameters to project these points onto the image. Of course, the two conversion relationships can also be integrated in advance to obtain the conversion relationship between the radar point cloud coordinate system and the image coordinate system. In summary, the conversion relationship described in this embodiment can be understood as the conversion relationship between the image coordinate system and the radar point cloud coordinate system, and can also be understood as the conversion relationship between the camera coordinate system and the radar point cloud coordinate system, or information containing these two conversion relationships.

[0120] Exemplarily, the stitching lines, endpoints and intersections of the stitching lines in the image data are identified and located to obtain the expression and point coordinates of the line in the image coordinate system; then, based on the conversion relationship between the image coordinate system and the camera coordinate system, the point coordinates in the image coordinate system are converted into the point coordinates in the camera coordinate system, and / or the expression of the line in the image coordinate system is converted into the expression of the line in the camera coordinate system.

[0121] Step 304: Determine the conversion relationship between the radar point cloud coordinate system and the camera coordinate system according to the first information and the second information.

[0122] In this embodiment, Figure 1 The target shown is used as an example for illustration. The conversion relationship can be calculated based on any point of L1, L2, L3, L4, L5, and L6, for example, the conversion relationship is calculated using the coordinates of the three intersection points. The calculation process is similar to the calibration method in the prior art and will not be repeated here.

[0123] In this embodiment, a three-dimensional point cloud of each plane on the target is obtained by radar, and image data of the target is obtained by camera; wherein, the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes; thus, a three-dimensional point cloud can be obtained by radar, and compared with a two-dimensional calibration object, the detected point cloud is obtained based on distance, and does not need to be inferred based on parameters such as reflectivity, so the accuracy of subsequent conversion relationships can be improved. The three-dimensional point cloud is plane-fitted to obtain a fitting plane for characterizing each plane of the target in the radar point cloud coordinate system, and based on the fitting plane, first information is determined, and the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system; thus, the plane equation of the target can be obtained according to the three-dimensional point cloud fitting, and the intersection points, endpoints, line equations, etc. of the splicing line can be calculated by plane equations, which is convenient for subsequent calculation of conversion relationships between different coordinate systems. Identify the second information in the image data, the second information is used to characterize the position and / or posture of the calibration reference object in the camera coordinate system; thereby, the intersection points, endpoints, line equations, etc. corresponding to the aforementioned three-dimensional point cloud can be obtained, which are used to achieve accurate point / line conversion. According to the first information and the second information, the conversion relationship between the radar point cloud coordinate system and the camera coordinate system is determined. Thus, the position of each plane of the target can be accurately reflected by the three-dimensional point cloud collected by the radar. Since the plane on which the conversion relationship depends is fitted, there is no need to use discrete point clouds to distinguish the dividing line, thereby effectively improving the calibration accuracy between the radar point cloud coordinate system and the camera coordinate system.

[0124] In another exemplary embodiment, Figure 4 As shown, a joint calibration method of radar and camera is provided, and the method is applied to Figure 1 The target in is taken as an example to illustrate, including the following steps 401 to 406. Among them:

[0125] Step 401, obtaining three-dimensional point clouds of each plane on the target through a radar, and obtaining image data of the target through a camera.

[0126] Step 402 , performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in the radar point cloud coordinate system, and determining first information based on the fitting plane.

[0127] For the specific implementation process and technical effects of steps 401 to 402 in this embodiment, please refer to Figure 3 The descriptions of steps 301 to 302 are not repeated here.

[0128] Step 403 , evaluating the fitting accuracy according to the fitting structural parameters associated with the fitting intersection points and / or fitting intersection lines and the actual structural parameters of the target known in advance, to obtain an evaluation result.

[0129] The conversion relationship is determined when the evaluation result meets a preset standard.

[0130] In an exemplary embodiment, the actual structural parameters include a first distance between designated intersections in the target; the designated intersections are formed by the intersection of three designated planes in the target; the fitting structural parameters include a second distance between fitting intersections for characterizing the designated intersections in the radar point cloud coordinate system; each of the fitting intersections is formed by the intersection of three fitting planes; the evaluation result includes a distance error between the first distance and the second distance; and / or:

[0131] The actual structural parameters include a first angle between two designated planes spliced ​​together in the target, the fitting structural parameters include a second angle between the fitting planes spliced ​​together, and the evaluation result includes an angle error between the first angle and the second angle.

[0132] In this embodiment, since the actual structural parameters of the target are fixed after the target is manufactured, some actual structural parameters can be used as a basis for evaluating the overall error (i.e., fitting accuracy). Figure 1 Taking the target shown as an example, the lengths of the stitching lines L1, L2, and L3, the angles between the surfaces, and the angles between the lines are known. Based on these known results, the results obtained after plane fitting can be evaluated.

[0133] Step 404, determine whether the evaluation result meets the requirements, if yes, execute step 405, if not, adjust the position and posture of the target and return to step 401.

[0134] In an exemplary embodiment, if the distance error is greater than a first preset threshold, the target's position and posture are adjusted and the three-dimensional point cloud is re-collected by radar, and plane fitting is performed based on the re-collected three-dimensional point cloud until the distance error between the intersection points of the planes is no greater than the first preset threshold.

[0135] In this embodiment, the distance between the intersections of each plane is determined according to the mathematical expression of each plane, and the distance error between the intersections of each plane is evaluated according to the actual structural parameters of the target known in advance. If the distance error is greater than a first preset threshold, the position of the target is adjusted and the three-dimensional point cloud is re-collected by radar, and plane fitting is performed based on the re-collected three-dimensional point cloud until the distance error between the intersections of each plane is no greater than the first preset threshold, and the mathematical expression of each plane of the target is obtained.

[0136] For example, Figure 1 Taking the target shown as an example, since the lengths of L1, L2, and L3 are determined, after obtaining (x1, y1), (x2, y2), and (x3, y3), the distances between the intersection points can be calculated to obtain three distances D1, D2, and D3; L1, L2, L3 are compared with D1, D2, and D3 to calculate the overall error (such as the average value and variance of the difference, which can be understood as the fitting accuracy); after knowing the fitting accuracy, it can also include: if the overall error is small (for example, less than a threshold), it is determined that the conversion relationship can be calculated using (x1, y1), (x2, y2), and (x3, y3). If the overall error is large, re-detection, re-fitting, and re-calculation can be performed. It is also possible to re-detect, re-fit, and re-calculate after adjusting the position and posture of the target, or to adjust (x1, y1), (x2, y2), and (x3, y3).

[0137] For example, if any one of the difference between L1 and D1, the difference between L2 and D2, and the difference between L3 and D3 is greater than the threshold, it is considered that the error is large and requires re-detection, fitting, calculation, or adjustment of (x1, y1), (x2, y2), and (x3, y3). Otherwise, the conversion relationship can be calculated using (x1, y1), (x2, y2), and (x3, y3).

[0138] If (x1, y1), (x2, y2), and (x3, y3) are adjusted, the adjustment process is as follows:

[0139] In the first case (fitting to the D plane first, and then determining the A, B, and C planes in turn based on the D plane), the plane fitting results of the A, B, and C planes are often more accurate. The fitting result of the D plane is slightly less accurate because the points near the boundary line are not eliminated and weighted. Based on this, it can be considered that the linear equations of L4, L5, and L6 calculated based on the A, B, and C planes are also relatively accurate. Therefore, when adjusting (x1, y1), (x2, y2), and (x3, y3), points can be taken based on the linear equations of L4, L5, and L6. One point is taken on each line each time to form a combination, that is, each combination contains three coordinates, recorded as (x1', y1'), (x2', y2'), and (x3', y3'). After taking points many times, several combinations can be obtained. For each combination, the corresponding distances between the three points are calculated, recorded as D1', D2', and D3', and the overall error between them and L1, L2, and L3 is calculated. Finally, a combination with the smallest error is taken as the new (x1, y1), (x2, y2), and (x3, y3), and then subsequent processing is performed based on this.

[0140] Exemplarily, other points can be taken within a certain radius of the three points (x1, y1), (x2, y2), and (x3, y3). These points can be points on the A, B, C, and D planes. Of course, they may not be or not entirely on the A, B, C, and D planes. Each time a group of points is taken, a combination of (x1", y1"), (x2", y2"), and (x3", y3") can be obtained. For each combination, the distances between the corresponding three points are calculated, recorded as D1", D2", and D3", and the overall error between them and L1, L2, and L3 is calculated. Finally, the combination with the smallest error is taken as the new (x1, y1), (x2, y2), and (x3, y3), and subsequent processing is performed accordingly.

[0141] In another exemplary embodiment, if the angle error is greater than a second preset threshold, the target's position is adjusted and the three-dimensional point cloud is collected again by radar, and plane fitting is performed according to the re-collected three-dimensional point cloud until the angle error between the planes is no greater than the second preset threshold, and the fitting planes of the target planes are obtained.

[0142] In this embodiment, the angles between the planes are determined based on the mathematical expressions of the planes, and the angle errors between the planes are evaluated based on the pre-known structural parameters of the target. If the angle error is greater than a second preset threshold, the position of the target is adjusted and the three-dimensional point cloud is re-collected by radar, and plane fitting is performed based on the re-collected three-dimensional point cloud until the angle error between the planes is no greater than the second preset threshold, and the mathematical expressions of the planes of the target are obtained.

[0143] For example, Figure 1 Taking the target shown in the figure as an example, since the angles between the faces have been determined, the angles between the fitted planes A, B, C, and D can be calculated and compared with the actual angles to obtain the corresponding angle errors. Then, the fitting accuracy can be calculated based on the angle errors. The four faces can form 6 angles, and the angles between the four simulated planes can be recorded as: θ AB ,θ AC ,θ AD ,θ BC ,θ BD ,θ CD The actual angle can be written as: β AB , β AC , β AD , β BC , β BD , β CD ; The difference between the corresponding angles can be calculated, for example, θ AB With β ABIf any difference is greater than the second preset threshold, the error is considered too large and re-detection, re-fitting, and re-calculation can be performed. Alternatively, the target position can be adjusted and re-detected, re-fitted, and re-calculated. If all differences are less than the second preset threshold, the error is considered small and (x1, y1), (x2, y2), and (x3, y3) are calculated, and the conversion relationship is calculated accordingly.

[0144] It should be noted that this embodiment is not limited to the above two methods, and the two methods can also be combined. For example, any one of the difference between L1 and D1, the difference between L2 and D2, and the difference between L3 and D3 is greater than the threshold A, or: θ AB With β AB The difference, θ AC With β AC The difference, θ AD With β AD The difference, θ BC With β BC The difference, θ BD With β BD The difference, θ CD With β CD If any one of the differences is greater than the second preset threshold, it is necessary to re-detect, fit, calculate, or make adjustments. Otherwise, the conversion relationship can be calculated based on (x1, y1), (x2, y2), and (x3, y3).

[0145] For example, Figure 1 Taking the target shown as an example, after obtaining the plane equations of the four planes A, B, C, and D, the plane equations can be combined (any three plane equations are linked to obtain the corresponding line equations) to obtain the line equations of L1, L2, L3, L4, L5, and L6. In the first case (fitting to the D plane first, and then determining the A, B, and C planes in turn based on the D plane), the plane fitting results of the A, B, and C planes are often more accurate. Based on this, it can also be considered that the linear equations of L4, L5, and L6 calculated based on the A, B, and C planes are also relatively accurate. Then, since the lengths of L1, L2, and L3 are known, we can directly select points on L4, L5, and L6 based on the lengths and linear equations to form combinations, that is, each combination contains three coordinates, recorded as (x1', y1'), (x2', y2'), and (x3', y3'). For each combination, calculate the corresponding distances between the three points, recorded as D1', D2', and D3', calculate the overall error with L1, L2, and L3, and finally take the combination with the smallest error as the new (x1, y1), (x2, y2), and (x3, y3), and then perform subsequent conversion relationship calculations based on this.

[0146] In this embodiment, there is no need to use the above method to obtain (x1, y1), (x2, y2), and (x3, y3) only when the overall error is large. Instead, the lengths of L1, L2, L3 and the linear equations of L4, L5, and L6 are directly used as constraints to find (x1, y1), (x2, y2), and (x3, y3).

[0147] Step 405: Identify second information in the image data.

[0148] Step 406: Determine the conversion relationship between the radar point cloud coordinate system and the camera coordinate system according to the first information and the second information.

[0149] For the specific implementation process and technical effects of steps 405 to 406 in this embodiment, please refer to Figure 3 The descriptions of steps 303 to 304 are not repeated here.

[0150] In this embodiment, the accuracy of plane fitting is evaluated by using the structural parameters of the target known in advance to obtain an evaluation result, and then based on the evaluation structure, it is determined whether re-fitting is needed (it can be simple re-detection and calculation, or it can be re-fitting and calculation after adjusting the target posture) to improve the accuracy of the point coordinates and / or line expressions in the radar point cloud coordinate system as much as possible, thereby improving the accuracy of the final conversion relationship.

[0151] In yet another exemplary embodiment, Figure 5 As shown, a joint calibration method of radar and camera is provided, and the method is applied to Figure 1 The target in is taken as an example to illustrate, including the following steps 501 to 506. Among them:

[0152] Step 501, obtaining three-dimensional point clouds of each plane on the target through a radar, and obtaining image data of the target through a camera.

[0153] Step 502 , performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in the radar point cloud coordinate system, and determining first information based on the fitting plane.

[0154] Step 503: Identify second information in the image data.

[0155] Step 504: Determine a conversion relationship between the radar point cloud coordinate system and the camera coordinate system according to the first information and the second information.

[0156] For the specific implementation process and technical effects of steps 501 to 504 in this embodiment, please refer to Figure 3 The descriptions of steps 301 to 304 are not repeated here.

[0157] Step 505 , according to the conversion relationship between the radar point cloud coordinate system and the camera coordinate system, project some or all of the points in the three-dimensional point cloud into the camera coordinate system, and compare the deviation between the projection result and the original reference object in the image.

[0158] In this embodiment, the deviation includes: the position of the point, and / or the angle between the lines. Figure 1 Taking the target shown as an example, after projecting the points of the stitching line L1 to the coordinate system of the camera, in order to distinguish them, the original line L1 in the image under the coordinate system is recorded as: L1_camera, and the line obtained after projecting the points of L1 is recorded as: L1 rader.

[0159] Step 506: Adjust the conversion relationship between the radar point cloud coordinate system and the camera coordinate system according to the deviation.

[0160] In this embodiment, referring to the embodiment of step 505, further, the deviation between L1_camera and L1 rader (for example, at least one of the following: angle, difference in endpoint positions, etc.) may be calculated, and the conversion relationship may be corrected accordingly.

[0161] In this embodiment, by comparing the deviation between the projection result and the original reference object in the image, the obtained conversion relationship is adjusted, which can further improve the accuracy of the conversion relationship.

[0162] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0163] Based on the same inventive concept, the embodiment of the present application also provides a radar and camera joint calibration device for implementing the radar and camera joint calibration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more radar and camera joint calibration devices provided below can refer to the limitations of the radar and camera joint calibration method above, and will not be repeated here.

[0164] In an exemplary embodiment, Figure 6 As shown, a joint calibration device for radar and camera is provided, including: an acquisition module 601, a fitting module 602, a positioning module 603 and a conversion module 604, wherein:

[0165] The acquisition module 601 is used to obtain the three-dimensional point cloud of each plane on the target through the radar, and to obtain the image data of the target through the camera; wherein the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and the spliced ​​planes form a splicing line, and the splicing line and / or the points on the splicing line are used as calibration reference objects;

[0166] A fitting module 602 is used to perform plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system, and determine first information based on the fitting plane, where the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system;

[0167] A positioning module 603, configured to identify second information in the image data, wherein the second information is used to characterize the position and / or posture of the calibration reference object in a camera coordinate system;

[0168] The conversion module 604 is used to determine the conversion relationship between the radar point cloud coordinate system and the camera coordinate system according to the first information and the second information.

[0169] Exemplarily, the calibration reference object includes at least one of the following:

[0170] A designated intersection point formed by the common intersection of three designated planes in the target;

[0171] The calibration reference object includes a designated stitching line formed by two designated planes in the target;

[0172] The endpoints of the designated splicing line;

[0173] The designated points for dividing the stitching line equally;

[0174] If the calibration reference object includes the designated intersection point, the first information includes the coordinates of the fitting intersection point used to characterize the designated intersection point in the radar point cloud coordinate system;

[0175] If the calibration reference object includes the designated stitching line, the first information includes a line equation for representing a fitting intersection line of the designated stitching line in the radar point cloud coordinate system;

[0176] If the calibration reference object includes the endpoints of the designated stitching line, the first information includes the coordinates of the endpoints of the fitted intersection line representing the designated stitching line in the radar point cloud coordinate system;

[0177] If the calibration reference object includes equally divided points of the designated stitching line, the first information includes coordinates of equally divided points of the fitted intersection line representing the designated stitching line in the radar point cloud coordinate system.

[0178] Exemplarily, the fitting module 602 is also used to evaluate the fitting accuracy based on the fitting structural parameters associated with the fitting intersection point and / or the fitting intersection line, and the actual structural parameters of the target known in advance, to obtain an evaluation result; the conversion relationship is determined when the evaluation result meets the preset standard.

[0179] Exemplarily, the actual structural parameters include a first distance between designated intersections in the target; the designated intersections are formed by the intersection of three designated planes in the target; the fitting structural parameters include a second distance between fitting intersections used to characterize the designated intersections in the radar point cloud coordinate system; each of the fitting intersections is formed by the intersection of three fitting planes; the evaluation result includes a distance error between the first distance and the second distance;

[0180] and / or:

[0181] The actual structural parameters include a first angle between two designated planes spliced ​​together in the target, the fitting structural parameters include a second angle between the fitting planes spliced ​​together, and the evaluation result includes an angle error between the first angle and the second angle.

[0182] Exemplarily, the at least three planes in the target include a first plane and multiple second planes, the first plane is polygonal, each of the second planes is spliced ​​to a straight edge of the first plane, the second planes formed by two adjacent straight edges of the first plane are also spliced ​​to each other, and the angle between the first plane and the second plane is an obtuse angle.

[0183] Exemplarily, the fitting module 602 is specifically used to: determine at least three points manually selected in the three-dimensional point cloud, and the at least three points are determined as points used to represent the first plane in the radar point cloud coordinate system; based on the at least three points, fit a first fitting plane used to represent the first plane; determine a bounding box of the points corresponding to the first fitting plane, the shape of the bounding box is the same as the first plane, and each edge of the bounding box corresponds to an edge of the first plane; select a group of point clouds that do not belong to the first fitting plane within a specified range near each edge of the bounding box, and based on each selected group of point clouds, fit a second fitting plane used to represent a second plane.

[0184] Exemplarily, the fitting module 602 is specifically used to: randomly select a number of points in the target point cloud in the three-dimensional point cloud, and then fit the point cloud in the target point cloud based on the number of points; if the plane cannot be fitted, re-select a number of points; if the plane can be fitted, determine that the fitted plane is the fitting plane, filter out the points corresponding to the fitting plane in the target point cloud, and use the remaining point cloud as the new target point cloud; repeat the above process to obtain the fitting plane used to characterize each plane of the target.

[0185] Exemplarily, the fitting module 602 is specifically configured to perform any of the following methods:

[0186] Method 1: clustering the three-dimensional point cloud to obtain multiple clusters, wherein the three-dimensional point cloud in each cluster corresponds to a plane; selecting a cluster as a target cluster, performing plane fitting on the three-dimensional point cloud in the target cluster to obtain a first fitting plane; respectively determining the distance between each point in the three-dimensional point cloud of the remaining other clusters and the first fitting plane, and screening the three-dimensional point clouds in the other clusters according to the distance to obtain a target point cloud; performing plane fitting on the target point cloud to obtain a second fitting plane corresponding to the other clusters; wherein the farther the distance between the three-dimensional point cloud in the other cluster and the first fitting plane is, the higher the probability of being selected as the target point cloud;

[0187] Method 2: clustering the three-dimensional point cloud to obtain multiple clusters, wherein the three-dimensional point cloud in each cluster corresponds to a plane; selecting a cluster as a target cluster, performing plane fitting on the three-dimensional point cloud in the target cluster to obtain a first fitting plane; respectively determining the distance between each point in the three-dimensional point cloud of the remaining other clusters and the first fitting plane, and dividing the other clusters into at least two point cloud groups according to the distance; respectively performing plane fitting on the three-dimensional point cloud in each point cloud group to obtain a plane equation of a sub-fitting plane corresponding to each point cloud group; weighted summing the coefficients of the plane equation of the sub-fitting plane corresponding to each point cloud group to obtain a plane equation of a second fitting plane corresponding to other clusters, wherein the weighted weight of the sub-fitting plane corresponding to the point cloud group that is closer to the first fitting plane is lower.

[0188] In an optional embodiment, the above-mentioned device may also include: an evaluation and adjustment module 605, wherein the evaluation and adjustment module 605 is used to: when the distance error is greater than a first preset threshold, adjust the position of the target and re-collect the three-dimensional point cloud through the radar, and perform plane fitting according to the re-collected three-dimensional point cloud until the distance error between the intersection points of each plane is not greater than the first preset threshold; and / or,

[0189] When the angle error is greater than a second preset threshold, the target's position and posture are adjusted and the three-dimensional point cloud is collected again by radar, and plane fitting is performed based on the re-collected three-dimensional point cloud until the angle error between the planes is no greater than the second preset threshold, so as to obtain the fitting planes of the target's planes.

[0190] Exemplarily, the evaluation and adjustment module 605 is also used for: after performing plane fitting on the three-dimensional point cloud to obtain the fitting planes for characterizing each plane of the target in the radar point cloud coordinate system, evaluating the fitting accuracy according to the fitting structural parameters associated with the fitting intersection points and / or the fitting intersection lines, and the actual structural parameters of the target known in advance, to obtain an evaluation result; the conversion relationship is determined when the evaluation result meets the preset standard.

[0191] Exemplarily, the actual structural parameters include a first distance between designated intersections in the target; the designated intersections are formed by the intersection of three designated planes in the target; the fitting structural parameters include a second distance between fitting intersections used to characterize the designated intersections in the radar point cloud coordinate system; each of the fitting intersections is formed by the intersection of three fitting planes; the evaluation result includes a distance error between the first distance and the second distance;

[0192] And / or: the actual structural parameters include a first angle between two designated planes spliced ​​together in the target, the fitting structural parameters include a second angle between the fitting planes spliced ​​together, and the evaluation result includes an angle error between the first angle and the second angle.

[0193] Exemplarily, the at least three planes in the target include a first plane and a plurality of second planes, the first plane is polygonal, each of the second planes is spliced ​​to a straight edge of the first plane, the second planes formed by two adjacent straight edges of the first plane are also spliced ​​to each other, and the angle between the first plane and the second plane is an obtuse angle; wherein the first plane intersects with any two second planes to form a designated intersection, and the calibration reference object includes at least three designated intersections; the three-dimensional point cloud of the target cluster corresponds to the first plane, and the three-dimensional point clouds of the other clusters correspond to the second plane;

[0194] Based on the fitting planes, determining the first information includes: based on the fitting intersection line between the second fitting planes and the pre-known length of the stitching line between the second planes in the target, determining the coordinates of the fitting intersection point used to characterize the designated intersection point in the radar point cloud coordinate system as part or all of the first information.

[0195] In another optional embodiment, the above-mentioned device may also include: a comparison and adjustment module 606, wherein the comparison and adjustment module 606 is used to: project part or all of the points in the three-dimensional point cloud to the coordinate system of the camera according to the conversion relationship between the radar point cloud coordinate system and the camera coordinate system, and compare the deviation between the projection result and the original reference object in the image, wherein the deviation includes: the position of the point, and / or the angle between the lines; and adjust the conversion relationship between the radar point cloud coordinate system and the camera coordinate system according to the deviation.

[0196] Each module in the above-mentioned radar and camera joint calibration device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0197] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a joint calibration method of a radar and a camera is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0198] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0199] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0200] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0201] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0202] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0203] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0204] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0205] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A joint calibration method for radar and camera, characterized in that: The method comprises: Acquire three-dimensional point clouds of each plane on the target through a radar, and acquire image data of the target through a camera; wherein the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes, and the splicing line and / or points on the splicing line are used as calibration reference objects; Performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system, and determining first information based on the fitting plane, where the first information is used to characterize the position and / or posture of the calibration reference object in the radar point cloud coordinate system; identifying second information in the image data, the second information being used to characterize a position and / or a posture of the calibration reference object in a camera coordinate system; A conversion relationship between a radar point cloud coordinate system and a camera coordinate system is determined according to the first information and the second information.

2. The method according to claim 1, characterized in that The calibration reference object includes at least one of the following: A designated intersection point formed by the common intersection of three designated planes in the target; The calibration reference object includes a designated stitching line formed by two designated planes in the target; The endpoints of the designated splicing line; The designated points for dividing the stitching line equally; If the calibration reference object includes the designated intersection point, the first information includes the coordinates of the fitting intersection point used to characterize the designated intersection point in the radar point cloud coordinate system; If the calibration reference object includes the designated stitching line, the first information includes a line equation for representing a fitting intersection line of the designated stitching line in the radar point cloud coordinate system; If the calibration reference object includes the endpoints of the designated stitching line, the first information includes the coordinates of the endpoints of the fitted intersection line representing the designated stitching line in the radar point cloud coordinate system; If the calibration reference object includes equally divided points of the designated stitching line, the first information includes coordinates of equally divided points of the fitted intersection line representing the designated stitching line in the radar point cloud coordinate system.

3. The method according to claim 2, characterized in that After performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in the radar point cloud coordinate system, the method further includes: The fitting accuracy is evaluated according to the fitting structural parameters associated with the fitting intersection point and / or the fitting intersection line, and the actual structural parameters of the target known in advance, to obtain an evaluation result; the conversion relationship is determined when the evaluation result meets the preset standard.

4. The method according to claim 3, characterized in that The actual structural parameters include a first distance between designated intersections in the target; the designated intersections are formed by the intersection of three designated planes in the target; the fitting structural parameters include a second distance between fitting intersections used to characterize the designated intersections in the radar point cloud coordinate system; each of the fitting intersections is formed by the intersection of three fitting planes; the evaluation result includes a distance error between the first distance and the second distance; and / or: The actual structural parameters include a first angle between two designated planes spliced ​​together in the target, the fitting structural parameters include a second angle between the fitting planes spliced ​​together, and the evaluation result includes an angle error between the first angle and the second angle.

5. The method according to claim 4, characterized in that The method further comprises: If the distance error is greater than a first preset threshold, the target's position and posture are adjusted and the three-dimensional point cloud is collected again by the radar, and plane fitting is performed according to the re-collected three-dimensional point cloud until the distance error between the intersection points of each plane is no greater than the first preset threshold; and / or, If the angle error is greater than a second preset threshold, the target's position and posture are adjusted and the three-dimensional point cloud is collected again by radar, and plane fitting is performed based on the re-collected three-dimensional point cloud until the angle error between the planes is no greater than the second preset threshold, so as to obtain the fitting planes of the target's planes.

6. The method according to any one of claims 1 to 5, characterized in that: The at least three planes in the target include a first plane and multiple second planes, the first plane is polygonal, each of the second planes is spliced ​​to a straight edge of the first plane, and the second planes formed by two adjacent straight edges of the first plane are also spliced ​​to each other, and the angle between the first plane and the second plane is an obtuse angle.

7. The method according to claim 6, characterized in that The performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system includes: Determining at least three manually selected points in the three-dimensional point cloud, wherein the at least three points are determined as points used to represent the first plane in the radar point cloud coordinate system; Based on the at least three points, obtain a first fitting plane for representing the first plane by fitting; Determine a bounding box of the point corresponding to the first fitting plane, wherein the shape of the bounding box is the same as that of the first plane, and each side of the bounding box corresponds to an side of the first plane; A group of point clouds not belonging to the first fitting plane is selected within a specified range near each side of the bounding box, and a second fitting plane for representing a second plane is fitted based on each selected group of point clouds.

8. The method according to any one of claims 1 to 5, characterized in that: The performing plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system includes: Randomly select a number of points in the target point cloud in the three-dimensional point cloud, and then perform fitting based on the number of points and the point cloud in the target point cloud; If the plane cannot be fitted, select several new points; If a plane can be fitted, the fitted plane is determined to be the fitting plane, points corresponding to the fitting plane are screened out from the target point cloud, and the remaining point cloud is used as a new target point cloud; The above process is repeated to obtain a fitting plane for characterizing each plane of the target.

9. A radar and camera joint calibration device, characterized in that: The device comprises: An acquisition module, used to obtain three-dimensional point clouds of each plane on the target through a radar, and to obtain image data of the target through a camera; wherein the target is composed of at least three planes spliced ​​at an angle other than 180 degrees, and a splicing line is formed between the spliced ​​planes, and the splicing line and / or points on the splicing line are used as calibration reference objects; a fitting module, configured to perform plane fitting on the three-dimensional point cloud to obtain a fitting plane for characterizing each plane of the target in a radar point cloud coordinate system, and determine first information based on the fitting plane, wherein the first information is used to characterize a position and / or posture of the calibration reference object in the radar point cloud coordinate system; a positioning module, configured to identify second information in the image data, wherein the second information is used to characterize the position and / or posture of the calibration reference object in a camera coordinate system; A conversion module is used to determine a conversion relationship between a radar point cloud coordinate system and a camera coordinate system according to the first information and the second information.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images