Radar and camera calibration method and device, electronic equipment and storage medium
Through the point cloud acquisition and transformation matrix correction methods of radar and camera, efficient and accurate calibration of radar and camera is achieved, solving the problems of low calibration efficiency and low accuracy in the prior art, and is suitable for a variety of environments.
Patent Information
- Application Number
- CN202411111921.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-12-06
AI Technical Summary
The existing radar and camera calibration methods are inefficient and low in accuracy, are easily restricted by the environment, and need to be performed in open and debris-free places. Manually selecting radar point clouds increases randomness errors.
The calibration plate is collected and extracted by radar, and the first coarse transformation matrix of the calibration plate coordinate system to the radar coordinate system is determined through the normal vector and the center point of the plane point cloud, and the calibration is carried out in combination with the boundary point cloud to obtain the accurate transformation matrix. , convert the coordinates of the graph code endpoint to the radar coordinate system through the precise transformation matrix, and determine the transformation matrix of the radar coordinate system to the camera coordinate system based on the actual distance of the lightning reaching the camera, so as to realize automatic calibration.
It improves the accuracy and efficiency of radar and camera calibration, reduces manual intervention, can calibrate in various environments, and reduces the complexity of the calibration process.
Smart Images

Figure CN119087409A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of sensor fusion calibration technology, and in particular to a radar and camera calibration method, device, electronic equipment and storage medium. Background Art
[0002] In the field of mobile robots, the positioning and navigation of robots are related to visual sensors (cameras) and laser sensors (radars). The robot moves autonomously in the environment, detects environmental information through its own sensors, records the contour information of the environment and finally generates a map or obstacle information.
[0003] The calibration of radar and camera can directly affect the quality of mapping and the extraction of perception information. Especially in the field of autonomous driving, the accurate calibration of cameras and lidar is particularly critical. Therefore, the calibration methods of radar and camera have become a hot spot and difficulty in the field of mobile robots in recent years. The current calibration methods are prone to the following problems:
[0004] 1. Extracting radar point clouds requires manual selection, which increases the complexity of the calibration process and also introduces random errors caused by manual selection;
[0005] 2. The accuracy of calibration is limited by the environment and needs to be in an open and clutter-free place;
[0006] In general, the current radar and camera calibration methods have the problems of low radar and camera calibration efficiency and low accuracy. Summary of the invention
[0007] The present invention provides a radar and camera calibration method to solve the problems of low radar and camera calibration efficiency and low precision in current radar and camera calibration methods.
[0008] In a first aspect, the present invention provides a radar and camera calibration method, comprising:
[0009] The radar is used to collect and extract point clouds of the calibration plate to obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate;
[0010] Determine a first coarse transformation matrix from a calibration plate coordinate system to a radar coordinate system based on a normal vector and a center point of the planar point cloud;
[0011] Correcting the first coarse transformation matrix by combining the first boundary point cloud and a second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system;
[0012] The calibration endpoint coordinates of the code endpoint on the calibration plate in the calibration plate coordinate system are transformed into the radar coordinate system by the precise transformation matrix to obtain the radar endpoint coordinates;
[0013] Determine a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual distance from the radar to the camera;
[0014] The second coarse transformation matrix is corrected by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
[0015] In a second aspect, the present invention provides a radar and camera calibration device, comprising:
[0016] A point cloud acquisition and extraction module is used to acquire and extract point clouds from the calibration plate using a radar, and obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate;
[0017] A first coarse transformation matrix determination module, used to determine a first coarse transformation matrix from a calibration plate coordinate system to a radar coordinate system based on a normal vector and a center point of the plane point cloud;
[0018] A precise transformation matrix determination module, configured to correct the first rough transformation matrix by combining the first boundary point cloud and a second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system, to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system;
[0019] A radar endpoint coordinate calculation module is used to transform the calibration endpoint coordinates of the code endpoint on the calibration plate in the calibration plate coordinate system to the radar coordinate system through the precise transformation matrix to obtain the radar endpoint coordinates;
[0020] A second coarse transformation matrix determination module, used to determine a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual object distance from the radar to the camera;
[0021] The calibration module is used to correct the second coarse transformation matrix by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
[0022] In a third aspect, the present invention provides an electronic device, the electronic device comprising:
[0023] at least one processor; and
[0024] a memory communicatively connected to the at least one processor; wherein,
[0025] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the radar and camera calibration method described in the first aspect of the present invention.
[0026] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the radar and camera calibration method described in the first aspect of the present invention when executed by a processor.
[0027] An embodiment of the present invention provides a radar and camera calibration method, comprising: using a radar to collect and extract point clouds on a calibration plate to obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate; determining a first coarse transformation matrix from a calibration plate coordinate system to a radar coordinate system based on a normal vector and a center point of the plane point cloud; correcting the first coarse transformation matrix in combination with the first boundary point cloud and a second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system; transforming the calibration endpoint coordinates of a code endpoint on the calibration plate in the calibration plate coordinate system to the radar coordinate system through the precise transformation matrix to obtain the radar endpoint coordinates; determining a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual distance from the radar to the camera; correcting the second coarse transformation matrix in combination with the radar endpoint coordinates and the camera endpoint coordinates of the code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system. First, a first coarse transformation matrix is obtained according to the normal vector and the center point of the plane point cloud, and then the first coarse transformation matrix is corrected in combination with the second boundary point cloud in the calibration plate coordinate system to obtain a precise transformation matrix. The precise transformation matrix and the calibration endpoint coordinates of the calibration plate are both of high precision. When the calibration endpoint coordinates are converted to the radar coordinate system through the precise transformation matrix, the precise radar endpoint coordinates can also be obtained. In the subsequent process, when the calibration transformation matrix from the radar coordinate system to the camera coordinate system is calculated based on the radar endpoint coordinates, the accuracy of the calibration transformation matrix can be improved. In addition, the radar and camera calibration method is automated, and there is no need for manual screening of point clouds, which improves the calibration efficiency.
[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0030] Figure 1 is a flow chart of a radar and camera calibration method provided in Embodiment 1 of the present invention;
[0031] Figure 2 It is a comparison diagram of a first rough transformation matrix and an accurate transformation matrix provided in the first embodiment of the present invention when transforming the boundary of the calibration plate;
[0032] Figure 3 It is a boundary schematic diagram of a calibration plate fitted based on a first boundary point cloud provided in the first embodiment of the present invention;
[0033] Figure 4 This is a schematic diagram of a calibration plate provided in Embodiment 1 of the present invention;
[0034] Figure 5 is a flow chart of a radar and camera calibration method provided in Embodiment 2 of the present invention;
[0035] Figure 6 is a schematic diagram of point clouds in different scanning forms provided in the second embodiment of the present invention;
[0036] Figure 7 This is a schematic diagram of plane point cloud segmentation provided by Embodiment 2 of the present invention;
[0037] Figure 8 is a structural schematic diagram of a radar and camera calibration device provided in Embodiment 3 of the present invention;
[0038] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION
[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0040] Embodiment 1
[0041] Figure 1This is a flow chart of a radar and camera calibration method provided in Embodiment 1 of the present invention. This embodiment is applicable to radar and camera calibration. The method can be performed by a radar and camera calibration device. The radar and camera calibration device can be implemented in the form of hardware and / or software. The radar and camera calibration device can be configured in an electronic device. Figure 1 As shown, the radar and camera calibration method includes:
[0042] S101 . Collect and extract point clouds of a calibration plate using a radar to obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate.
[0043] LiDAR acquires three-dimensional information of the surrounding environment by emitting laser beams and measuring the reflection time, while the camera provides visible light images. The main purpose of joint calibration of radar and camera is to improve the visualization and integrity of LiDAR point cloud and enhance data association and analysis capabilities. Taking autonomous vehicles as an example, the vehicle obtains information about the surrounding environment through radar and camera. Radar and camera are two different devices, usually located in different directions. When the two work at the same time, the direction information of the data obtained by the two is inconsistent. In order to produce more accurate and consistent perception results, radar and camera are calibrated in the subsequent process.
[0044] The calibration plate is used to determine the conversion relationship between physical size and pixels in applications such as machine vision, image measurement, and photogrammetry. In the present invention, the camera and the radar are located at different positions relative to the calibration plate, but the calibration plate coordinate system where the calibration plate is located is known and fixed, and the calibration plate can be used as an intermediate reference to calibrate the camera and the radar.
[0045] The calibration plate usually has a specific pattern array or marker, and the specific pattern array or marker is located on the plane of the calibration plate. The planar point cloud of the plane of the calibration plate can be quickly extracted through the specific pattern array or marker.
[0046] The first boundary point cloud corresponding to the boundary of the calibration plate is extracted based on the plane point cloud. Specifically, the plane point cloud can be projected into the radar coordinate system, and edge detection can be performed on the plane point cloud to identify the boundary position of the calibration plate, thereby obtaining the first boundary point cloud at the boundary position.
[0047] S102: Determine a first coarse transformation matrix from a calibration plate coordinate system to a radar coordinate system based on a normal vector and a center point of the plane point cloud.
[0048] Specifically, it includes: obtaining the normal vector of the plane where the radar is located; obtaining the rotation matrix based on the normal vector of the plane point cloud and the normal vector of the plane where the radar is located; taking the center point of the plane point cloud as the translation matrix; and obtaining the first coarse transformation matrix from the calibration plate coordinate system to the radar coordinate system based on the rotation matrix and the translation matrix.
[0049] The center point of the plane point cloud can be specifically obtained by respectively calculating the mean values of the plane point cloud in the X-axis, Y-axis, and Z-axis directions as the three-dimensional coordinate value of the center point.
[0050] When calibrating the rotation between the radar and the calibration plate, the plane in the radar and the plane in the calibration plate are extracted for alignment, and the overlap information between the normal of the radar plane and the plane of the calibration plate is found to calculate the rotation matrix of the calibration plate under the radar.
[0051] Two normal vectors can be used to calculate the rotation matrix, which can transform one vector into another. This is especially true when the rotation relationship between two coordinate systems needs to be determined. Usually, the plane normal in one coordinate system is extracted, and then compared and calculated with the plane normal in another coordinate system to determine the rotation matrix. The normal direction can be calculated based on the plane point cloud. The radar plane normal vector can be (0, 0, 1). The radar plane normal vector is perpendicular to the ground.
[0052] The rotation matrix is used to describe the rotation operation of an object in space, and the translation matrix is used to describe the position movement of an object in space. By appropriately combining the rotation matrix and the translation matrix, a transformation matrix can be formed, which can simultaneously describe the rotation and translation movement of an object. This combination can be achieved by matrix multiplication, in which the rotation matrix acts before a point or vector, and the translation matrix acts on it, thereby obtaining the first coarse transformation matrix in this embodiment.
[0053] The first coarse transformation matrix is a coarse transformation matrix from the calibration plate coordinate system to the radar coordinate system. When the calibration plate is transformed into the radar coordinate system based on the first coarse transformation matrix, the obtained calibration plate posture is the same as the posture of the calibration plate obtained by the actual transformation (both in the radar coordinate system). Figure 2 This is a comparison diagram of the first rough transformation matrix and the accurate transformation matrix when transforming the calibration plate boundary, such as Figure 2 As shown, A is the boundary of the calibration plate obtained when the calibration plate is converted to the radar coordinate system based on the first coarse transformation matrix, and B is the boundary of the calibration plate obtained when the calibration plate boundary is actually converted to the radar coordinate system (based on the accurate transformation matrix). The two may not completely overlap, so it is necessary to further calculate the precise transformation matrix.
[0054] Optionally, before executing S102, the point cloud may be clustered by calling a clustering function of the PCL library to remove non-planar points, so as to screen the planar point cloud and obtain an accurate center point.
[0055] S103 , correcting the first coarse transformation matrix by combining the first boundary point cloud and the second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system.
[0056] The second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system can be obtained directly by measurement. The coordinate points of the square frame of the calibration plate in space (including four corner points and edge points) are directly set. Specifically, a corner point of the square frame of the calibration plate can be used as the coordinate origin. After determining the calibration plate coordinate system, the second boundary coordinate set of the boundary of the calibration plate in the calibration plate coordinate system, that is, the second boundary point cloud, can be directly measured.
[0057] like Figure 2 As shown, A is the boundary of the calibration plate in the calibration plate coordinate system, and B is the fitted boundary of the calibration plate obtained when the second boundary point cloud of the calibration plate is converted to the radar coordinate system based on the first coarse transformation matrix. The two may not completely overlap, so it is necessary to further calculate the precise transformation matrix.
[0058] The first boundary point cloud is in the radar coordinate system, so the first boundary point cloud can be inversely transformed through the first coarse transformation matrix to obtain the point cloud of the first boundary point cloud in the calibration plate coordinate system, and then compared with the second boundary point cloud according to the transformed point cloud to determine the matching effect between the two, and then the first coarse transformation matrix can be corrected according to the matching effect to obtain the precise transformation matrix from the calibration plate coordinate system to the radar coordinate system.
[0059] It is also possible to first perform frame fitting on the first boundary point cloud in the radar coordinate system to obtain the boundary posture of the calibration plate in the radar coordinate system. Figure 3 is a schematic diagram of the boundary of the calibration plate based on the first boundary point cloud fitting, such as Figure 3 As shown in FIG. 1 , s is the boundary of the calibration plate fitted in the radar coordinate system, and the points on or around the boundary A are the first boundary point cloud. After the fitting is completed, the boundary of the calibration plate fitted in the radar coordinate system is transformed to the calibration plate coordinate system through the first rough transformation matrix to match with the second boundary point cloud, and obtain the precise transformation matrix from the calibration plate coordinate system to the radar coordinate system.
[0060] Optionally, the first coarse transformation matrix is corrected by combining the first boundary point cloud and the second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system, including: using the first coarse transformation matrix to inversely transform the first boundary point cloud from the radar coordinate system to the calibration plate coordinate system to obtain a transformed boundary point cloud; matching the second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system with the transformed boundary point cloud to obtain a first intermediate transformation matrix of the second boundary point cloud relative to the transformed boundary point cloud; and calculating the product of the first coarse transformation matrix and the first intermediate transformation matrix to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system.
[0061] Among them, the matching process of the converted point cloud and the second boundary point cloud can be performed based on the PNP algorithm of OpenCV. It should also be noted that in the process of calculating the precise transformation matrix, the point cloud / coordinates in the radar coordinate system can be converted to the calibration plate coordinate system for unified comparison and matching. At this time, the inverse transformation is performed according to the first rough transformation matrix. The point cloud / coordinates in the calibration plate coordinates can also be converted to the radar coordinate system for unified comparison and matching. At this time, the transformation is performed according to the first rough transformation matrix. Both can achieve the purpose of obtaining a precise transformation matrix, and the present invention is not limited to this.
[0062] S104, transforming the calibration endpoint coordinates of the code endpoint on the calibration plate in the calibration plate coordinate system to the radar coordinate system through a precise transformation matrix to obtain the radar endpoint coordinates.
[0063] The calibration board is provided with a graphic code, which includes a non-linear QR code, a hand-painted code, a text pattern code, etc. Figure 4 As shown, Figure 4 It is a schematic diagram of a calibration plate. A plurality of code images (including code images 51, 52 and 53) are set in the calibration plate 50. The type of the code image is a two-dimensional code. The code image endpoints on the calibration plate 50 are the four corner points of the square frame where the code image is located. There are multiple code image endpoints. The calibration endpoint coordinates of the code image endpoints in the calibration plate coordinate system can also be directly measured. The calibration endpoint coordinates of the code image endpoints on the calibration plate in the calibration plate coordinate system are transformed to the radar coordinate system through the precise transformation matrix to obtain the radar endpoint coordinates.
[0064] S105 , determining a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual distance from the radar to the camera.
[0065] The camera is used as the origin of the camera coordinate system to measure the three-dimensional distance from the radar to the camera, and the x value, y value, and z value are obtained as the rough transformation matrix from the radar coordinate system to the camera coordinate system.
[0066] S106, correcting the second coarse transformation matrix by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
[0067] Just like radar collects data to obtain point clouds, the camera can also obtain the camera endpoint coordinates from the code endpoints detected by the camera.
[0068] Similarly, the second coarse transformation matrix is a coarse transformation matrix from the radar coordinate system to the camera coordinate system. When the radar endpoint coordinates collected (and converted) by the radar are transformed to the camera coordinate system based on the second coarse transformation matrix, there may be an orientation error.
[0069] The radar endpoint coordinates are the coordinates in the radar coordinate system. Therefore, the radar endpoint coordinates can be transformed by the second coarse transformation matrix to obtain the point cloud of the radar endpoint coordinates in the camera coordinate system. Then, the converted point cloud is compared with the camera endpoint coordinates to determine the matching effect between the two. Then, the second coarse transformation matrix can be corrected according to the matching effect to obtain the calibration transformation matrix from the radar coordinate system to the camera coordinate system.
[0070] Optionally, the second coarse transformation matrix is corrected in combination with the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system, including: using the second coarse transformation matrix to convert the radar endpoint coordinates from the radar coordinate system to the camera coordinate system to obtain transformation endpoint coordinates; matching the camera endpoint coordinates of the image code endpoint detected by the camera with the transformation endpoint coordinates to obtain a second intermediate transformation matrix of the transformation endpoint coordinates relative to the camera endpoint coordinates; calculating the product of the second coarse transformation matrix and the second intermediate transformation matrix to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
[0071] An embodiment of the present invention provides a radar and camera calibration method, comprising: using a radar to collect and extract point clouds on a calibration plate to obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate; determining a first coarse transformation matrix from a calibration plate coordinate system to a radar coordinate system based on a normal vector and a center point of the plane point cloud; correcting the first coarse transformation matrix in combination with the first boundary point cloud and a second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system; transforming the calibration endpoint coordinates of a code endpoint on the calibration plate in the calibration plate coordinate system to the radar coordinate system through the precise transformation matrix to obtain the radar endpoint coordinates; determining a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual distance from the radar to the camera; correcting the second coarse transformation matrix in combination with the radar endpoint coordinates and the camera endpoint coordinates of the code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system. First, a first coarse transformation matrix is obtained according to the normal vector and the center point of the plane point cloud, and then the first coarse transformation matrix is corrected in combination with the second boundary point cloud in the calibration plate coordinate system to obtain a precise transformation matrix. The precise transformation matrix and the calibration endpoint coordinates of the calibration plate are both of high precision. When the calibration endpoint coordinates are converted to the radar coordinate system through the precise transformation matrix, the precise radar endpoint coordinates can also be obtained. In the subsequent process, when the calibration transformation matrix from the radar coordinate system to the camera coordinate system is calculated based on the radar endpoint coordinates, the accuracy of the calibration transformation matrix can be improved. In addition, the radar and camera calibration method is automated, and there is no need for manual screening of point clouds, which improves the calibration efficiency.
[0072] Embodiment 2
[0073] Figure 5 This is a flow chart of a radar and camera calibration method provided in Embodiment 2 of the present invention. This embodiment of the present invention is optimized based on Embodiment 1 above. Figure 5 As shown, the radar and camera calibration method includes:
[0074] S501 , using a radar to collect and extract point clouds from a calibration plate, to obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate.
[0075] Optional, such as Figure 4As shown, a reflective strip 54 is also attached to the calibration plate 50. The calibration plate is collected and extracted by radar to obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to the boundary of the calibration plate, including: collecting data of the calibration plate by radar to obtain an initial point cloud; extracting points whose point cloud intensity is greater than a preset intensity threshold from the initial point cloud as reflective strip point clouds; performing range screening on the initial point cloud by using the center point of the reflective strip point cloud and a preset distance threshold to obtain a plane point cloud of the calibration plate plane, wherein the distance threshold is the length of the long side of the calibration plate; performing boundary feature extraction on the plane point cloud to obtain a first boundary point cloud corresponding to the boundary of the calibration plate.
[0076] The point cloud intensity value is usually represented by an 8-bit unsigned integer from 0 to 255. The larger the value, the stronger the reflective ability. As a common safety sign, the reflective strip has the strongest reflective ability on the calibration plate. Therefore, the preset intensity threshold can be set to 250, and points with point cloud intensity greater than 250 are extracted as reflective strip point clouds. The center point of the reflective strip point cloud is calculated, and the point clouds with a distance from the center point of the preset distance threshold are all taken as plane point clouds.
[0077] As a common safety sign, the reflective strip may have a certain elasticity and plasticity. Therefore, when subjected to external force, the points on it are prone to deformation. This deformation can be in the form of bending, stretching or compression. There are also some non-planar points (brackets), so the initial point cloud is clustered by calling the pcl library clustering function to obtain a planar point cloud.
[0078] By using simple calibration tools such as reflective strips, the position of the calibration plate plane can be quickly located, and the point cloud of non-calibrated plates can be automatically filtered out, which can not only improve the efficiency of radar and camera calibration, but also improve the accuracy of the data.
[0079] The point cloud scanned by the radar includes irregular point cloud and ring-shaped scanning radar point cloud, such as Figure 6 As shown, Figure 6 A schematic diagram of a point cloud with different scanning forms. Figure 6 The middle left picture shows an irregular point cloud, and the right picture shows a ring-shaped scanning radar point cloud.
[0080] When the point cloud is a ring-shaped scanning point cloud, a regular point cloud is obtained, and the first boundary point cloud can be directly obtained, such as Figure 6 As shown, the maximum Y-axis value point and the minimum Y-axis value point in each row of the point cloud are used as the first boundary point cloud corresponding to the boundary of the calibration plate.
[0081] When the point cloud is an irregular point cloud, boundary features are extracted from the plane point cloud to obtain a first boundary point cloud corresponding to the boundary of the calibration plate, including: determining the maximum Z-axis value point and the minimum Z-axis value point in the plane point cloud; partitioning the area where the plane point cloud is located in the Z-axis direction based on the maximum Z-axis value point, the minimum Z-axis value point and the preset partition height to obtain multiple sub-areas; for each sub-area, extracting points in the sub-area whose height from the bottom of the sub-area is less than a preset height threshold as points in the filtered sub-area; and using the maximum Y-axis value point and the minimum Y-axis value point in the filtered sub-area as the first boundary point cloud corresponding to the boundary of the calibration plate.
[0082] Figure 7 As shown, Figure 7 Schematic diagram of plane point cloud segmentation, where 71 is the boundary of the plane point cloud, 72 is a sub-region, the sub-region 71 is a rectangular frame, the height of the rectangular frame in the Z-axis direction is the height of the partition, when partitioning, the bottom 72 of the sub-region is a straight line, and the points in the sub-region whose height from the bottom of the sub-region 72 is less than the preset height threshold can be extracted. Points that are approximately parallel to the bottom straight line of the sub-region 72 can be extracted, and the combined effect of the points extracted from each sub-region is similar to Figure 6 It is similar to the right figure in , which is convenient for extracting the maximum Y-axis value points and the minimum Y-axis value points at both ends, where the preset height threshold can be set to 0.01m.
[0083] Of course, points in the sub-region whose height from the top of the sub-region is less than a preset height threshold may also be extracted, and the principle is the same as above.
[0084] S502: Determine a first coarse transformation matrix from the calibration plate coordinate system to the radar coordinate system based on the normal vector and the center point of the plane point cloud.
[0085] S503 , correcting the first coarse transformation matrix by combining the first boundary point cloud and the second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system.
[0086] S504, transforming the calibration endpoint coordinates of the code endpoint on the calibration plate in the calibration plate coordinate system to the radar coordinate system through a precise transformation matrix to obtain the radar endpoint coordinates.
[0087] S505 , determining a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual distance from the radar to the camera.
[0088] S506 , correcting the second coarse transformation matrix by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
[0089] S501-S506 are similar to S101-S106 in the first embodiment. For details, please refer to the relevant contents of S101-S106, which will not be described in detail here.
[0090] S507 , using the calibration transformation matrix to transform the radar endpoint coordinates into the camera coordinate system to obtain the inspection endpoint coordinates.
[0091] The calibration transformation matrix is used to transform the radar endpoint coordinates into the camera coordinate system for accurate comparison.
[0092] S508: Normalize the coordinates of the inspection endpoints.
[0093] Specifically, the x value and y value of the test endpoint coordinates are normalized. For each test endpoint coordinate, the x value and y value of the test endpoint coordinates are normalized: x = x / z, y = y / z. The x value and y value on the left side of the equal sign are the normalized values, and the x value and y value on the right side of the equal sign are the values before normalization.
[0094] S509, calculating the distance error between the inspection endpoint coordinates and the camera endpoint coordinates.
[0095] For each endpoint, calculate the distance error between the test endpoint coordinates and the camera endpoint coordinates, that is, find the spatial distance between the two coordinates. The distance error calculation formula is:
[0096]
[0097] Among them, x ri ,y ri 、z ri is the coordinate of the test endpoint corresponding to the i-th code endpoint, x ci ,y ci 、z ci is the camera endpoint coordinate corresponding to the i-th code endpoint, Di is the distance error between the inspection endpoint coordinate corresponding to the i-th code endpoint and the camera endpoint coordinate.
[0098] S510: Determine whether the mean of all distance errors is less than a preset error threshold.
[0099] The calculation formula is: n is the number of endpoints of the graph code, is the mean of all distance errors.
[0100] If yes, execute S511, if no, execute S512.
[0101] S511, determine that the calibration transformation matrix is a correct calibration result.
[0102] When the mean value of the distance error is less than the preset error threshold, it means that the calibration result has high accuracy and meets the requirements, so the calibration transformation matrix is determined to be the correct calibration result.
[0103] S512: Use the calibration transformation matrix to replace the second coarse transformation matrix.
[0104] After the second coarse transformation matrix is corrected, the accuracy of the obtained endpoint calibration transformation matrix must be higher than the accuracy of the second coarse transformation matrix, that is, the calibration effect is better. Therefore, even if the current calibration transformation matrix does not meet the calibration accuracy requirements and needs to be recalibrated, the calibration transformation matrix can be used to replace the second coarse transformation matrix to gradually improve the calibration accuracy.
[0105] After executing S512, the process returns to executing S506 to recalibrate the second coarse transformation matrix until the mean value of the distance error is less than a preset error threshold.
[0106] The above S507-S510 is a process of checking the calibration result. When the calibration accuracy meets the requirements, the calibration result is output. When there is a large error in the calibration, recalibration is performed.
[0107] In this embodiment, the initial point cloud is range-screened by the center point of the reflective strip point cloud and the preset distance threshold, so that the plane point cloud can be quickly screened out, thereby improving the calibration efficiency and accuracy. When the calibration transformation matrix is obtained, the calibration result is checked to ensure the accuracy of the calibration result.
[0108] In order to facilitate a systematic understanding of the entire process of the radar and camera calibration method of the present invention, the following example is first used for illustration, and the example mainly includes the following steps (only numbers without drawings):
[0109] S1. First, paste the code (QR code) and reflective strip on the calibration board, and collect the point cloud of the calibration board through radar. The size of the calibration board is 0.5m*0.5m.
[0110] S2. Extract the points of the calibration plate according to the intensity of the reflective strip points.
[0111] 1) Extract points with point cloud intensity greater than 250, where the intensity range of laser point cloud is 0 to 255, and the intensity of general objects is about 100;
[0112] 2) Calculate the center point of the reflective strip Centre_Reflect (x, y, z),
[0113] Centre_Reflect(x, y, z)=∑(xi, yi, zi) / k, where (i∈k);
[0114] k represents the number of points corresponding to reflective strips, and the average of all reflective strip points is calculated.
[0115] 3) The point at a distance of 0.5m from the center point Centre_Reflect of the reflective strip is used as the laser point on the plane of the calibration plate, i.e., the plane point cloud;
[0116] 4) Cluster the point cloud by calling the pcl library clustering function, extract the center point Centre_board of the plane and calculate the normal vector ceff of the plane;
[0117] 5) According to the filtered plane point cloud, calculate the normal vector of the center of the calibration plate and the plane, and calculate the center point Centre_board in the calibration plate coordinate system.
[0118] Centre_board (x, y, z) = ∑ (xi, yi, zi) / m, where (i∈m);
[0119] Where m is the number of plane point clouds after filtering. According to the normal vector ceff (rotation matrix, dimension 3x3) and Centre_board (translation, dimension 1x3), the rough transformation matrix T_Lidar2Board_1 (transformation matrix, dimension 4x4) from the calibration board to the radar coordinate system is obtained. This matrix preliminarily expresses the location of the center point of the calibration board, and the orientation of the plane is roughly the same as that of the calibration board.
[0120] S3: When the plane point cloud is an irregular scanning point cloud, it is divided into multiple sub-areas according to the height of the irregular scanning point cloud. When the plane point cloud is a ring scanning point cloud, S3 is skipped.
[0121] 1) Assume that the height difference of the partition range is 0.1 meters. By comparing the z-axis values of each point in the point cloud, record the highest point height of the calibration plate as H, the lowest point height as D, and the number of sub-areas as Area_num = (HD) / 0.1;
[0122] 2) Partition and store the points belonging to each sub-area into Area_num sub-areas, such as Figure 7 shown.
[0123] 3) In order to extract the boundary points, the nearest point to the bottom of the ion region is extracted in each sub-region (the distance does not exceed 0.01m) as the filtered sub-region.
[0124] 4) The points corresponding to the maximum and minimum y values of the point cloud in the filtered sub-region are the left and right boundary point clouds end_points in each sub-region, such as Figure 3 shown.
[0125] S4. According to the physical object, the point cloud map is designed as a boundary point cloud map of a 0.5m*0.5m square point cloud map, and the boundary point cloud extracted in the previous step is matched with the current boundary point cloud.
[0126] 1) First, the boundary point cloud end_point of the calibration board in the radar coordinate system is inversely transformed to the calibration board coordinate system through the transformation matrix T_Lidar2Board_1 to obtain the point cloud end_point_board;
[0127] 2) Extract the boundary point cloud end_point_board of the calibration board and the boundary point cloud map_point of the physical design point cloud map for point matching (in the calibration board coordinate system), and calculate the transformation matrix T_Lidar2Board_2 of map_point in the calibration board coordinate system relative to the extracted boundary point cloud end_point_board of the calibration board;
[0128] 3) Calculate the precise transformation matrix T_Lidar2Board of the calibration board relative to the radar posture. The formula is:
[0129] T_Lidar2Board=T_Lidar2Board_1*T_Lidar2Board_2.
[0130] S5. According to the precise transformation matrix T_Lidar2Board of the calibration board relative to the radar, the coordinates of the corner points of the code in the radar coordinate system are calculated. According to the rough relative transformation of the camera under the radar, the fitted radar QR code coordinates are calculated, and then the radar to camera calibration is performed.
[0131] 1) The coordinates of the corner point of the image code relative to the calibration plate aruco_point can be marked on the calibration plate with a ruler. The aruco_point point is transformed by the precise transformation matrix T_Board2Lidar to obtain the radar endpoint coordinates in the radar coordinate system aruco_point_lidar = T_Lidar2Board*aruco_point;
[0132] 2) According to the physical measurement, the second rough transformation matrix T_Lidar2Camera_1 from radar to camera is obtained, and the radar endpoint coordinates aruco_point_lidar are transformed to obtain the point cloud aruco_point_lidar_in_camera of the approximate initial position of the radar QR code corner point in the camera. The formula is:
[0133] aruco_point_lidar_in_camera=T_Lidar2Camera_1*aruco_point_lidar.
[0134] 3) Use the camera to identify the coordinates of the corner points of the code to obtain the camera endpoint coordinates aruco_point_camera, and the initial position point cloud of the radar endpoint coordinates scanned by the radar in the camera coordinate system
[0135] aruco_point_lidar_in_camera and the camera endpoint coordinates aruco_point_camera monitored by the camera, call opencv's pnp matching, calculate the exact matching radar to camera coordinate system transformation matrix T_Lidar2Camera_2, and obtain the complete radar to camera transformation matrix, that is, the calibration transformation matrix: T_Lidar2Camera = T_Lidar2Camera_1*T_Lidar2Camera_2.
[0136] S6. Set the calculation matching score method to measure the calibration accuracy.
[0137] 1) The radar endpoint coordinates aruco_point_lidar in the radar coordinate system are transformed by T_Lidar2Camera to obtain the radar scanned QR code corner points aruco_point_lidar_in_cam in the camera coordinate system.
[0138] 2) The distance difference between the aruco_point_lidar_in_cam point cloud and the camera endpoint coordinates aruco_point_camera identified by the camera is calculated one by one, and the average distance error of each point is calculated. If the threshold is reached, the matching result is considered to be the correct calibration matrix. Otherwise, S5 is re-executed, but the initial matrix T_Lidar2Camera_1 needs to be updated. T_Lidar2Camera_1 is replaced by T_Lidar2Camera of S5 until the distance error is less than the threshold 1, and the calibration result is output.
[0139] The steps to calculate the error are as follows:
[0140] First, transform the radar point cloud to the camera coordinate system, and then normalize the X and Y coordinates of the transformed point cloud: X = X / Z, Y = Y / Z;
[0141] Update to each point, where X, Y, and Z are the x, y, and z axis coordinates of the point respectively, and the left side of the equal sign is the updated coordinate. Then rotate the radar normalized point to the camera's image coordinate system to obtain the new image coordinate (x i ,y i , z i), where i represents the serial number of the point cloud point. Finally, the absolute distance of the pixel is calculated, and the distance error Err_i is calculated by one-to-one correspondence between the point cloud point and the QR code point of the camera, where i represents the serial number of the endpoint of the code, and then all distance errors are accumulated to get the average.
[0142] Embodiment 3
[0143] Figure 8 This is a schematic diagram of the structure of a radar and camera calibration device provided in Embodiment 3 of the present invention. Figure 8 As shown, the radar and camera calibration device includes:
[0144] The point cloud acquisition and extraction module 801 is used to acquire and extract point clouds from the calibration plate using a radar, and obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate;
[0145] A first coarse transformation matrix determining module 802 is used to determine a first coarse transformation matrix from the calibration plate coordinate system to the radar coordinate system based on the normal vector and the center point of the plane point cloud;
[0146] The precise transformation matrix determination module 803 is used to correct the first rough transformation matrix by combining the first boundary point cloud and the second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system;
[0147] The radar endpoint coordinate calculation module 804 is used to transform the calibration endpoint coordinates of the code endpoint on the calibration plate in the calibration plate coordinate system to the radar coordinate system through the precise transformation matrix to obtain the radar endpoint coordinates;
[0148] A second coarse transformation matrix determination module 805 is used to determine a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual distance from the radar to the camera;
[0149] The calibration module 806 is used to correct the second coarse transformation matrix by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
[0150] Optionally, a reflective strip is attached to the calibration plate. The point cloud acquisition and extraction module 801 includes:
[0151] The initial point cloud acquisition submodule is used to collect data from the calibration plate using radar to obtain the initial point cloud;
[0152] A reflective strip point cloud extraction submodule is used to extract points whose point cloud intensity is greater than a preset intensity threshold from the initial point cloud as reflective strip point clouds;
[0153] A plane point cloud screening submodule, used to perform range screening on the initial point cloud based on the center point of the reflective strip point cloud and a preset distance threshold to obtain a plane point cloud of the calibration plate plane, wherein the distance threshold is the length of the long side of the calibration plate;
[0154] The first boundary point cloud determination submodule is used to extract boundary features from the plane point cloud to obtain a first boundary point cloud corresponding to the boundary of the calibration plate.
[0155] Optionally, the first boundary point cloud determination submodule includes:
[0156] A height value determination unit, used to determine a maximum Z-axis value point and a minimum Z-axis value point in the plane point cloud;
[0157] A partitioning unit, configured to partition the area where the plane point cloud is located in the Z-axis direction based on the maximum Z-axis value point, the minimum Z-axis value point and a preset partition height to obtain a plurality of sub-areas;
[0158] An endpoint value extraction unit, used for extracting, for each of the sub-regions, points in the sub-region whose height from the bottom of the sub-region is less than a preset height threshold, as points in the sub-region after filtering;
[0159] The first boundary point cloud determining unit is used to use the maximum Y-axis value point and the minimum Y-axis value point in the filtered sub-area as the first boundary point cloud corresponding to the boundary of the calibration plate.
[0160] Optionally, the first coarse transformation matrix determining module 802 includes:
[0161] A first normal vector obtaining submodule is used to obtain a normal vector of the plane where the radar is located;
[0162] A rotation matrix obtaining submodule, used for obtaining a rotation matrix based on the normal vector of the plane point cloud and the normal vector of the plane where the radar is located;
[0163] A translation matrix obtaining submodule, used for taking the center point of the plane point cloud as a translation matrix;
[0164] The first coarse transformation matrix determination submodule is used to obtain a first coarse transformation matrix from the calibration plate coordinate system to the radar coordinate system based on the rotation matrix and the translation matrix.
[0165] Optionally, the precise transformation matrix determination module 803 includes:
[0166] A first conversion submodule, configured to use the first coarse transformation matrix to inversely transform the first boundary point cloud from a radar coordinate system to a calibration plate coordinate system to obtain a transformed boundary point cloud;
[0167] A first intermediate transformation matrix determination submodule is used to match a second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system with the transformed boundary point cloud to obtain a first intermediate transformation matrix of the second boundary point cloud relative to the transformed boundary point cloud;
[0168] The precise transformation matrix determination submodule is used to calculate the product of the first coarse transformation matrix and the first intermediate transformation matrix to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system.
[0169] Optionally, the calibration module 806 includes:
[0170] A second conversion submodule, configured to convert the radar endpoint coordinates from a radar coordinate system to a camera coordinate system using the second coarse transformation matrix to obtain transformed endpoint coordinates;
[0171] A second intermediate transformation matrix determination submodule is used to match the camera endpoint coordinates of the image code endpoint detected by the camera with the transformation endpoint coordinates to obtain a second intermediate transformation matrix of the transformation endpoint coordinates relative to the camera endpoint coordinates;
[0172] The calibration submodule is used to calculate the product of the second coarse transformation matrix and the second intermediate transformation matrix to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
[0173] Optional radar and camera calibration equipment, including:
[0174] A test endpoint coordinate acquisition module, used to convert the radar endpoint coordinates into a camera coordinate system using the calibration transformation matrix to obtain the test endpoint coordinates;
[0175] A normalization module, used for normalizing the coordinates of the inspection endpoints;
[0176] A distance error calculation module, used to calculate the distance error between the inspection endpoint coordinates and the camera endpoint coordinates;
[0177] A distance error judgment module is used to judge whether the mean of all the distance errors is less than a preset error threshold; if so, the content of the calibration result confirmation module is executed; if not, the content of the update module is executed;
[0178] A calibration result confirmation module, used to determine that the calibration transformation matrix is a correct calibration result;
[0179] The updating module is used to replace the second coarse transformation matrix with the calibration transformation matrix, and return the content of the calibration module 806.
[0180] The radar and camera calibration device provided in the embodiment of the present invention can execute the radar and camera calibration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0181] Embodiment 4
[0182] Fig. 9 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0183] like Fig. 9 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.
[0184] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0185] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 41 performs the various methods and processes described above, such as radar and camera calibration methods.
[0186] In some embodiments, the radar and camera calibration method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the radar and camera calibration method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the radar and camera calibration method in any other suitable manner (e.g., by means of firmware).
[0187] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0188] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0189] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0190] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0191] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0192] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0193] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0194] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A radar and camera calibration method, characterized in that: include: The radar is used to collect and extract point clouds of the calibration plate to obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate; Determine a first coarse transformation matrix from a calibration plate coordinate system to a radar coordinate system based on a normal vector and a center point of the planar point cloud; Correcting the first coarse transformation matrix by combining the first boundary point cloud and a second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system; The calibration endpoint coordinates of the code endpoint on the calibration plate in the calibration plate coordinate system are transformed into the radar coordinate system by the precise transformation matrix to obtain the radar endpoint coordinates; Determine a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual distance from the radar to the camera; The second coarse transformation matrix is corrected by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
2. The radar and camera calibration method according to claim 1, characterized in that: A reflective strip is attached to the calibration plate. The radar is used to collect and extract point clouds from the calibration plate to obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate, including: The radar is used to collect data from the calibration plate to obtain the initial point cloud; Extracting points whose point cloud intensity is greater than a preset intensity threshold from the initial point cloud as reflective strip point clouds; The initial point cloud is range-screened using the center point of the reflective strip point cloud and a preset distance threshold to obtain a plane point cloud of the calibration plate plane, wherein the distance threshold is the length of the long side of the calibration plate; Boundary features are extracted from the plane point cloud to obtain a first boundary point cloud corresponding to the boundary of the calibration plate.
3. The radar and camera calibration method according to claim 2, characterized in that: The step of extracting boundary features from the plane point cloud to obtain a first boundary point cloud corresponding to the boundary of the calibration plate includes: Determine the maximum Z-axis value point and the minimum Z-axis value point in the plane point cloud; Partitioning the area where the plane point cloud is located in the Z-axis direction based on the maximum Z-axis value point, the minimum Z-axis value point and a preset partition height to obtain a plurality of sub-areas; For each of the sub-regions, extracting points in the sub-region whose height from the bottom of the sub-region is less than a preset height threshold as points in the sub-region after filtering; The maximum Y-axis value point and the minimum Y-axis value point in the filtered sub-area are used as the first boundary point cloud corresponding to the boundary of the calibration plate.
4. The radar and camera calibration method according to claim 1, characterized in that: The first coarse transformation matrix from the calibration plate coordinate system to the radar coordinate system is determined based on the normal vector and the center point of the plane point cloud, comprising: Get the normal vector of the plane where the radar is located; Obtaining a rotation matrix based on the normal vector of the plane point cloud and the normal vector of the plane where the radar is located; Taking the center point of the plane point cloud as a translation matrix; A first coarse transformation matrix from the calibration plate coordinate system to the radar coordinate system is obtained based on the rotation matrix and the translation matrix.
5. The radar and camera calibration method according to claim 1, characterized in that: The step of correcting the first coarse transformation matrix by combining the first boundary point cloud and the second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system includes: Using the first coarse transformation matrix, the first boundary point cloud is inversely transformed from the radar coordinate system to the calibration plate coordinate system to obtain a transformed boundary point cloud; Matching a second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system with the transformed boundary point cloud to obtain a first intermediate transformation matrix of the second boundary point cloud relative to the transformed boundary point cloud; The product of the first coarse transformation matrix and the first intermediate transformation matrix is calculated to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system.
6. The radar and camera calibration method according to claim 1, characterized in that: The step of correcting the second coarse transformation matrix by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system includes: The radar endpoint coordinates are converted from a radar coordinate system to a camera coordinate system using the second coarse transformation matrix to obtain transformed endpoint coordinates; Matching the camera endpoint coordinates of the code endpoint detected by the camera with the transformation endpoint coordinates to obtain a second intermediate transformation matrix of the transformation endpoint coordinates relative to the camera endpoint coordinates; The product of the second coarse transformation matrix and the second intermediate transformation matrix is calculated to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
7. The radar and camera calibration method according to any one of claims 1 to 6, characterized in that: After obtaining the calibration transformation matrix from the radar coordinate system to the camera coordinate system, it also includes: The calibration transformation matrix is used to transform the radar endpoint coordinates into a camera coordinate system to obtain the inspection endpoint coordinates; Normalizing the inspection endpoint coordinates; Calculating the distance error between the inspection endpoint coordinates and the camera endpoint coordinates; Determine whether the mean of all the distance errors is less than a preset error threshold; If so, determining that the calibration transformation matrix is a correct calibration result; If not, the calibration transformation matrix is used to replace the second coarse transformation matrix, and the step of correcting the second coarse transformation matrix by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera is returned to obtain the calibration transformation matrix from the radar coordinate system to the camera coordinate system.
8. A radar and camera calibration device, characterized in that: include: A point cloud acquisition and extraction module is used to acquire and extract point clouds from the calibration plate using a radar, and obtain a plane point cloud of the calibration plate plane and a first boundary point cloud corresponding to a boundary of the calibration plate; A first coarse transformation matrix determination module, used to determine a first coarse transformation matrix from a calibration plate coordinate system to a radar coordinate system based on a normal vector and a center point of the plane point cloud; A precise transformation matrix determination module, configured to correct the first rough transformation matrix by combining the first boundary point cloud and a second boundary point cloud of the boundary of the calibration plate in the calibration plate coordinate system, to obtain a precise transformation matrix from the calibration plate coordinate system to the radar coordinate system; A radar endpoint coordinate calculation module is used to transform the calibration endpoint coordinates of the code endpoint on the calibration plate in the calibration plate coordinate system to the radar coordinate system through the precise transformation matrix to obtain the radar endpoint coordinates; A second coarse transformation matrix determination module, used to determine a second coarse transformation matrix from the radar coordinate system to the camera coordinate system according to the actual object distance from the radar to the camera; The calibration module is used to correct the second coarse transformation matrix by combining the radar endpoint coordinates and the camera endpoint coordinates of the image code endpoint detected by the camera to obtain a calibration transformation matrix from the radar coordinate system to the camera coordinate system.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the radar and camera calibration method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the radar and camera calibration method according to any one of claims 1 to 7 when executed.
Citation Information
Cited By
Indoor positioning precision evaluation method and device based on visual two-dimensional code
CN120800436A
