Radar and camera joint calibration method and device, computer equipment and storage medium
By forming a common field of view in the multi-radar and camera system and collecting calibration data, and using the calibration plate point cloud extraction features for calibration, the efficiency and accuracy problems of joint calibration of multi-lidar and cameras are solved, and efficient calibration process and accuracy are achieved. The coordinate conversion relationship is determined.
Patent Information
- Application Number
- CN202411005630.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art is difficult to effectively improve the joint calibration efficiency of multi-lidar and cameras while ensuring accuracy, especially in large-scale mass production, calibration accuracy and efficiency are difficult to improve simultaneously.
By adjusting the joint system of multi-radar and the camera, a common field of view is formed, and the calibration object and calibration plate are placed in the area, the calibration data of the radar and camera are collected, and the calibration data is extracted by the calibration plate point cloud to achieve the calibration between the radar and camera. The external parameters and internal parameters are calibrated simultaneously to improve the efficiency and accuracy of external parameters calibration.
On the premise of ensuring accuracy, the calibration process is simplified, the calibration efficiency between multiple sensors is improved, the demand for large-scale mass production is met, and the user experience is improved.
Smart Images

Figure CN118967828A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multi-sensor calibration, and in particular to a radar and camera joint calibration method, device, computer equipment and storage medium. Background Art
[0002] With the rapid development of autonomous driving and assisted driving, environmental perception technology is particularly important, and the use of laser radar and industrial cameras for multi-sensor fusion is the current mainstream environmental perception solution. Accurately perceiving the environment around the vehicle can timely and accurately discover safety hazards during driving, and then quickly take safety measures to avoid traffic accidents. For fusion perception, the premise is to complete the fusion at the data level, that is, to find the corresponding relationship between each laser radar point cloud and between the camera and the laser radar point cloud. Therefore, in most autonomous driving solutions, it is necessary to first complete the spatial coordinate unification of multiple sensors and determine the coordinate transformation relationship of each sensor data space. The data obtained by each laser radar is a three-dimensional point cloud with the same data structure. The external parameter calibration between multiple laser radars is mainly to determine the rotation matrix and translation vector. For the calibration of laser radar and camera, it is necessary to complete the conversion relationship between three-dimensional coordinates and two-dimensional coordinates. Therefore, in addition to determining the external parameters between the laser radar and camera coordinate systems, namely the rotation matrix and translation vector, it is also necessary to determine the internal parameters of the camera to complete the transformation between three-dimensional coordinates and two-dimensional coordinates.
[0003] In related technologies, the joint calibration methods of cameras and lidars can be roughly divided into feature-based methods and optimization-based methods, but most calibration algorithms solve external parameters for a single lidar and camera, focusing on improving calibration accuracy. However, with the increasing application of multi-sensor systems, it is difficult to effectively improve the calibration efficiency of multiple lidars and cameras while ensuring accuracy in large-scale mass production. Summary of the invention
[0004] In view of this, the present invention provides a radar and camera joint calibration method, device, computer equipment and storage medium to solve the problem that it is difficult to effectively improve the calibration efficiency of multiple laser radars and cameras while ensuring accuracy.
[0005] In a first aspect, the present invention provides a radar and camera joint calibration method, which is applied to a joint system composed of several radars and cameras, and the method includes:
[0006] Adjust the relative positions of the radars and cameras to form a common field of view between the radars and cameras;
[0007] Calibration objects and calibration plates are set at target positions within the public field of view, and image data and point cloud data of key data frames of the calibration objects and calibration plates at different target positions are collected;
[0008] Based on the position of the calibration plate in the image data of the key data frame, the intrinsic parameters of the camera are calibrated to obtain the camera coordinate system;
[0009] Based on the point cloud data corresponding to the calibration object in the key data frame, the external parameters of the radar are calibrated to obtain the radar coordinate system;
[0010] Based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system, the camera and radar are jointly calibrated to obtain the conversion relationship between the camera coordinate system and the radar coordinate system.
[0011] In the present invention, a common field of view area is formed by adjusting the joint system composed of multiple radars and cameras, and calibration objects and calibration plates are placed in the common field of view area. The data of the calibration objects and calibration plates in the radar and camera are collected, and the camera and radar are calibrated, so that the external and internal parameter calibration between radar and radar and between radar and camera is completed at the same time. By collecting calibration data in the same calibration scene by multiple sensors of radar and camera at the same time and completing the calibration based on the feature extraction of the point cloud of the calibration plate, the external parameter calibration efficiency and calibration accuracy between multiple sensors can be improved, the calibration process is simplified, and it is helpful to meet the needs of large-scale mass production. Multiple laser radars and cameras are effectively combined under the premise of ensuring accuracy, the calibration efficiency is improved, and the user experience is further improved.
[0012] In an optional implementation, collecting image data and point cloud data of key data frames of the calibration object and the calibration plate at different target positions includes:
[0013] The point cloud data corresponding to the calibration object at different target positions is acquired by radar, and the image data corresponding to the calibration plate at different target positions is acquired by camera;
[0014] Collect radar timestamp data of the radar and camera timestamp data of the camera;
[0015] Match the radar timestamp data with the camera timestamp data on the time axis, and calculate the time difference between the radar timestamp data and the camera timestamp data;
[0016] Based on the time difference, the radar and camera are synchronized to match the point cloud data and image data at the same time;
[0017] Based on the point cloud data and image data at the same time, key data frames are screened and the image data and point cloud data of the key data frames are determined.
[0018] In this method, by time synchronizing the data collected by the radar and the camera, it is easy to select key data frames, which is beneficial for the subsequent use of the image data of the key data frames for camera calibration and the use of the point cloud data of the key data frames for radar calibration.
[0019] In an optional implementation, based on the point cloud data and image data at the same time, key data frames are screened and obtained, including:
[0020] Selecting the image data of the first data frame as the initial key frame, and calculating the structural similarity between the image data of each data frame and the image data of the adjacent data frames in sequence starting from the initial key frame;
[0021] Determine whether the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is greater than a preset threshold;
[0022] When the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is greater than a preset threshold, determining the current data frame as a key data frame;
[0023] When the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is less than a preset threshold, the image data of the current frame and the corresponding point cloud data are removed;
[0024] Traverse until the data frames corresponding to all image data are calculated.
[0025] In this method, by selecting key data frames and filtering out data of non-key data frames, the accuracy of the data is further improved, which is beneficial to improving the calibration efficiency and calibration accuracy.
[0026] In an optional implementation, the intrinsic parameters of the camera are calibrated based on the image data of the key data frame to obtain the camera coordinate system, including:
[0027] Based on the pixel coordinates of the calibration plate in the image data of the key data frame and the world coordinates of the calibration plate, a first conversion relationship between the camera's internal coordinate system and the world coordinate system is obtained;
[0028] Get the homography matrix constructed by multiplying the camera's intrinsic and extrinsic parameters;
[0029] The homography matrix and the first transformation relationship are solved jointly to obtain a second transformation relationship between the homography matrix and the intrinsic and extrinsic parameters of the camera;
[0030] Use the homography matrix to construct a symmetric matrix, solve the symmetric matrix, and obtain the intrinsic parameters of the camera;
[0031] Based on the intrinsic parameters of the camera and the second transformation relationship, the extrinsic parameters of the camera are solved, and the camera coordinate system is determined based on the intrinsic parameters and extrinsic parameters of the camera.
[0032] In this method, the image data of the calibration plate acquired by the camera and the coordinates of the calibration plate in the world coordinate system are used to solve the intrinsic and extrinsic parameters of the camera, which together constitute the camera coordinate system, thereby facilitating the joint calibration of the camera coordinate system and the radar coordinate system.
[0033] In an optional implementation, based on the point cloud data of the key data frame, the external parameters of the radar are calibrated to obtain the radar coordinate system, including:
[0034] Extract the calibration object point cloud from the point cloud data of the key data frame;
[0035] Set the main radar, use the radar coordinate system corresponding to the main radar as the target coordinate system, and set the initial radar extrinsics between the remaining radars and the main radar;
[0036] Based on the initial radar external parameters, the first point set of the calibration object point cloud in the main radar is registered with the second point set of the calibration object point cloud in the remaining radars to obtain a transformed point set;
[0037] Based on the transformation point set, an objective function is constructed, and the objective function is solved to obtain a third transformation relationship;
[0038] Performing coordinate transformation on the transformation point set based on the third transformation relationship to obtain a second transformation point set;
[0039] Based on the second transformation point set, the objective function is iteratively solved to obtain the radar coordinate system.
[0040] In this method, by setting the initial radar extrinsic parameters of the main radar and other radars to the main radar, the transformation point set is obtained by alignment, and the objective function is constructed. After the transformation point set is transformed, the objective function is iteratively solved using the transformed transformation point set to obtain the radar coordinate system, which is convenient for realizing the joint calibration of the camera coordinate system and the radar coordinate system.
[0041] In an optional implementation, based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system, the camera and the radar are jointly calibrated to obtain a conversion relationship between the camera coordinate system and the radar coordinate system, including:
[0042] Based on the coordinates of the calibration object in the camera coordinate system, obtain the normal vector of the calibration object in the camera coordinate system;
[0043] Establish the constraint of the calibration object on the normal vector in the camera coordinate system and obtain the first error function;
[0044] Based on the coordinates of the calibration object in the radar coordinate system and the first error function, a constraint relationship between the camera coordinate system and the radar coordinate system is established, and a conversion relationship between the camera coordinate system and the radar coordinate system is determined.
[0045] In this way, by using multiple sensors such as radar and camera to simultaneously collect calibration data in the same calibration scene and completing the calibration based on feature extraction based on the point cloud of the calibration plate, the efficiency and accuracy of external parameter calibration between multiple sensors can be improved, the calibration process can be simplified, and it helps to meet the needs of large-scale mass production, and effectively combine multiple laser radars and cameras while ensuring accuracy.
[0046] In a second aspect, the present invention provides a radar and camera joint calibration device, the device comprising:
[0047] A common field of view area forming module is used to adjust the relative posture of each radar and camera to form a common field of view area between the radar and the camera;
[0048] A data acquisition module is used to set calibration objects at target positions within the public field of view, and to collect image data and point cloud data of key data frames of the calibration objects at different target positions;
[0049] The camera coordinate system calibration module is used to calibrate the camera's internal parameters based on the image data of the key data frame to obtain the camera coordinate system;
[0050] The radar coordinate system calibration module is used to calibrate the external parameters of the radar based on the point cloud data of the key data frame to obtain the radar coordinate system;
[0051] The joint calibration module is used to perform joint calibration of the camera and the radar based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system, and obtain the conversion relationship between the camera coordinate system and the radar coordinate system.
[0052] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the radar and camera joint calibration method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.
[0053] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the radar and camera joint calibration method of the above-mentioned first aspect or any corresponding embodiment thereof.
[0054] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the radar and camera joint calibration method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0056] Figure 1 4 is a flow chart of a radar and camera joint calibration method according to an embodiment of the present invention.
[0057] Figure 2 It is a flowchart of a method for joint calibration of multiple laser radars and cameras according to an embodiment of the present invention.
[0058] Figure 3 is a structural schematic diagram of a laser radar and camera joint calibration system according to an embodiment of the present invention.
[0059] Figure 4 4 is a flow chart of another radar and camera joint calibration method according to an embodiment of the present invention.
[0060] Figure 5 4 is a flow chart of another radar and camera joint calibration method according to an embodiment of the present invention.
[0061] Figure 6 is a schematic diagram of joint calibration of a camera and a lidar according to an embodiment of the present invention.
[0062] Figure 7 4 is a schematic diagram of a checkerboard plane normal vector constraint in a camera coordinate system according to an embodiment of the present invention.
[0063] Figure 8 4 is a structural block diagram of a radar and camera joint calibration device according to an embodiment of the present invention.
[0064] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0066] In related technologies, the joint calibration methods of cameras and lidars can be roughly divided into feature-based methods and optimization-based methods, but most calibration algorithms solve external parameters for a single lidar and camera, focusing on improving calibration accuracy. However, with the increasing application of multi-sensor systems, it is difficult to effectively improve the calibration efficiency of multiple lidars and cameras while ensuring accuracy in large-scale mass production.
[0067] In order to solve the above problems, a radar and camera joint calibration method is provided in an embodiment of the present invention, which is used in a computer device. It should be noted that its execution subject can be a radar and camera joint calibration device, which can be implemented as part or all of a computer device through software, hardware, or a combination of software and hardware. The computer device can be a terminal, a client, or a server. The server can be a single server or a server cluster composed of multiple servers. The terminal in the embodiment of the present application can be a smart phone, a personal computer, a tablet computer, or other intelligent hardware devices. In the following method embodiments, the execution subject is a computer device as an example for explanation.
[0068] The computer device in this embodiment is suitable for the use scenario of joint calibration of a system composed of several laser radars and cameras. The present invention provides a radar and camera joint calibration method, which forms a common field of view by adjusting the joint system composed of multiple radars and cameras, places calibration objects and calibration plates in the common field of view, collects data of the calibration objects and calibration plates in the radar and camera, calibrates the camera and radar, and realizes the simultaneous completion of external parameter and internal parameter calibration between radar and radar and between radar and camera. By using multiple sensors of radar and camera to simultaneously collect calibration data in the same calibration scene and extract features based on the point cloud of the calibration plate to complete the calibration, the external parameter calibration efficiency and calibration accuracy between multiple sensors can be improved, the calibration process is simplified, and it helps to meet the needs of large-scale mass production. Multiple laser radars and cameras are effectively combined under the premise of ensuring accuracy, the calibration efficiency is improved, and the user experience is further improved.
[0069] According to an embodiment of the present invention, an embodiment of a radar and camera joint calibration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0070] In this embodiment, a radar and camera joint calibration method is provided, which can be used for the above-mentioned computer device. Figure 1 is a flow chart of a radar and camera joint calibration method according to an embodiment of the present invention. Figure 1As shown, the process includes the following steps:
[0071] Step S101, adjusting the relative posture of each radar and camera to form a common field of view area between the radar and the camera.
[0072] In one example, the radar point cloud and camera image are viewed on the host PC, and the installation positions of the radars and cameras are adjusted to ensure that the radar point cloud and the camera image have overlapping fields of view.
[0073] Step S102 , setting calibration objects and calibration plates at target positions within the public field of view, and collecting image data and point cloud data of key data frames of the calibration objects and calibration plates at different target positions.
[0074] In one example, two calibration objects and a calibration plate are placed in a common field of view, wherein the calibration plate is a checkerboard calibration plate. The posture of the checkerboard calibration plate is continuously adjusted, and the checkerboards are sequentially placed at multiple target positions in the common field of view. Multiple sets of key data frames captured by each lidar sensor and camera sensor at different target positions of the calibration plate are obtained.
[0075] Step S103, calibrating the intrinsic parameters of the camera based on the position of the calibration plate in the image data of the key data frame to obtain the camera coordinate system.
[0076] In one example, the intrinsic parameters of the camera are calibrated according to the acquired checkerboard image data.
[0077] Step S104, based on the point cloud data corresponding to the calibration object in the key data frame, calibrate the external parameters of the radar to obtain the radar coordinate system.
[0078] In one example, the ICP point cloud registration algorithm is used to calibrate each lidar external parameter based on the collected multiple sets of calibration object data frames.
[0079] Step S105 , based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system, perform joint calibration of the camera and the radar to obtain a conversion relationship between the camera coordinate system and the radar coordinate system.
[0080] In one example, the camera intrinsic parameters obtained by calibration according to the above steps are used to perform joint calibration of the camera and the lidar according to the coordinate representation of the checkerboard plane in the camera coordinate system and the lidar coordinate system.
[0081] In one implementation scenario, Figure 2 FIG. 1 is a flow chart of a method for joint calibration of multiple laser radars and cameras according to an embodiment of the present invention. Figure 2As shown, the method for joint calibration of multiple laser radars and cameras includes: step S1, adjusting the relative posture of each laser radar and camera, and setting a common field of view area between each laser radar sensor and camera sensor. Step S2, placing two calibration objects and a calibration plate in the common field of view area, wherein the calibration plate is a checkerboard calibration plate; continuously adjusting the checkerboard posture, placing the checkerboard in turn at multiple target positions in the common field of view area; obtaining multiple sets of key data frames captured by each laser radar sensor and camera sensor at different target positions of the calibration plate. Step S3, calibrating the camera's internal parameters according to the collected checkerboard image data. Step S4, calibrating the external parameters of each laser radar using the ICP point cloud registration algorithm according to the collected multiple sets of calibration object data frames. Step S5, obtaining the camera's internal parameters obtained in step S3, and performing joint calibration of the camera and laser radar according to the coordinate representation of the checkerboard plane in the camera coordinate system and the laser radar coordinate system.
[0082] In another implementation scenario, Figure 3 is a schematic diagram of the structure of a laser radar and camera joint calibration system according to an embodiment of the present invention. Figure 3 As shown, the system includes: an acquisition module for acquiring the point cloud data of the laser radar and the image data of the camera. A data preprocessing module for preprocessing the collected sensor data, including filtering the point cloud, removing the ground point cloud, extracting the ROI area, extracting the calibration target surface and calibrating the camera internal parameters. A laser radar joint calibration module for jointly calibrating multiple laser radars, obtaining the external parameters between multiple laser radars, and determining the coordinate transformation relationship between the coordinate systems of each laser radar. A camera and laser radar joint module for jointly calibrating the camera and the laser radar to obtain the transformation relationship between the three-dimensional coordinates of the point cloud and the two-dimensional coordinates of the image. A coordinate system integration module completes the final calibration integration based on the external parameters between each laser radar and the external parameters between the camera and the main laser radar, determines the coordinate system transformation relationship between all sensors, and establishes a unified coordinate system.
[0083] Specifically, the data preprocessing module includes: a camera calibration unit, which is used to calibrate the camera's internal parameters and obtain the camera's internal parameters. A point cloud preprocessing unit, which is used to preprocess the point cloud data obtained by the laser radar, including: point cloud filtering, which removes outliers from the original point cloud data to reduce noise interference; ground point cloud removal: fitting and removing the ground point cloud through the RANSAC algorithm, further reducing the interference of irrelevant point clouds and improving the accuracy of point cloud matching. Point cloud clustering: The extraction of calibration target point clouds is completed through the Euclidean clustering algorithm, laying the foundation for the external parameter calibration between multiple laser radars. The key frame extraction unit is used to screen the data obtained by the laser radar and camera sensors, select key data frames with calibration value, and improve the accuracy of the calibration method.
[0084] The laser radar joint calibration module includes: a parameter initialization unit, which is used to initialize the multi-laser radar external parameter calibration method and provide initial values for the calibration algorithm. A multi-laser radar external parameter calibration unit is used to optimize and solve the external parameters between the multi-laser radars, and use the error function and optimization algorithm on the basis of the initial values to obtain the final conversion relationship between the multi-laser radar coordinate systems.
[0085] The camera and lidar joint module includes: the checkerboard extraction module in the image: used to calibrate the camera's external parameters, determine the conversion relationship between the camera coordinate system and the world coordinate system, and determine the mathematical representation of the checkerboard plane and the normal vector of the checkerboard plane in the camera coordinate system. The checkerboard extraction module in the point cloud: used to extract the checkerboard plane in the point cloud data obtained by the lidar, obtain the point cloud coordinates of the checkerboard plane, and provide point cloud parameters for the camera and lidar joint calibration algorithm. The external parameter solution module: used to construct the mathematical constraint relationship between the lidar and the camera, establish the error function through the mathematical constraint relationship, optimize and solve the error function, and finally obtain the external parameters between the camera and the lidar.
[0086] The radar and camera joint calibration method provided in this embodiment forms a common field of view by adjusting the joint system composed of multiple radars and cameras, placing calibration objects and calibration plates in the common field of view, collecting data of the calibration objects and calibration plates in the radar and camera, calibrating the camera and radar, and achieving the simultaneous completion of external and internal parameter calibration between radar and radar and between radar and camera. By collecting calibration data in the same calibration scene by multiple sensors of radar and camera at the same time and completing the calibration based on feature extraction from the point cloud of the calibration plate, the external parameter calibration efficiency and calibration accuracy between multiple sensors can be improved, the calibration process is simplified, and it helps to meet the needs of large-scale mass production. It effectively combines multiple laser radars and cameras while ensuring accuracy, improves calibration efficiency, and further improves the user experience.
[0087] In this embodiment, a radar and camera joint calibration method is provided, which can be used for the above-mentioned computer device. Figure 4 is a flowchart of another radar and camera joint calibration method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:
[0088] Step S401: adjust the relative positions of the radars and cameras to form a common field of view between the radars and cameras. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0089] Step S402 , setting calibration objects and calibration plates at target positions within the public field of view, and collecting image data and point cloud data of key data frames of the calibration objects and calibration plates at different target positions.
[0090] Specifically, the above step S402 includes:
[0091] Step S4021, using radar to collect point cloud data corresponding to the calibration object at different target positions, and using a camera to collect image data corresponding to the calibration plate at different target positions.
[0092] Step S4022, collecting radar timestamp data of the radar and camera timestamp data of the camera.
[0093] Step S4023, time axis matching the radar timestamp data and the camera timestamp data, and calculating the time difference between the radar timestamp data and the camera timestamp data.
[0094] Step S4024: Based on the time difference, the radar and the camera are synchronized to obtain the point cloud data and image data at the same time.
[0095] Step S4025, based on the point cloud data and image data at the same time, filter out key data frames, and determine the image data and point cloud data of the key data frames.
[0096] In some optional implementations, the above step S4025 includes:
[0097] Step a1: Select the image data of the first data frame as the initial key frame, and calculate the structural similarity between the image data of each data frame and the image data of the adjacent data frames in sequence starting from the initial key frame.
[0098] Step a2, determining whether the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is greater than a preset threshold.
[0099] Step a3: when the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is greater than a preset threshold, determining the current data frame as a key data frame.
[0100] Step a4: when the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is less than a preset threshold, the image data of the current frame and the corresponding point cloud data are removed.
[0101] Step a5, traverse until the data frames corresponding to all image data are calculated.
[0102] In one example, two calibration objects and a calibration plate are placed in a common field of view, wherein the calibration plate is a checkerboard calibration plate, and the calibration object must have a regular shape and high reflectivity. For example, a calibration rod with a reflectivity of 95%, a wavelength of 905nm, a height of 2m, and a diameter of 0.2m is selected as the calibration object. The calibration object is not limited in the present invention. The checkerboard posture is continuously adjusted, and the checkerboard is placed in a plurality of target positions in the common field of view in sequence, wherein the target position is the checkerboard placement position, and the checkerboard is placed in the overlapping field of view of the camera and the radar, and does not exceed the field of view distance of the camera and the radar. Multiple sets of key data frames captured by each lidar sensor and camera sensor at different target positions of the calibration plate are obtained. Specifically comprising the following steps:
[0103] Step S2.1: Obtain the timestamp data of the lidar and camera, match the timestamps of the lidar and camera through the time axis, and find their relative positions on the time axis.
[0104] Step S2.2: Determine the time difference between the camera and lidar data by calculating the difference in timestamps, and perform time synchronization by truncation. Time synchronization includes: when the camera and radar are calibrated together, it is necessary to ensure that the timestamps of the camera image data and the radar point cloud data match and align. The initial camera and radar time synchronization uses hardware synchronization. In order to ensure the error of the timestamp, by comparing the time difference between the camera and radar data, if a frame of data with the smallest time difference is selected, it is considered that the radar and camera data in this frame match and align.
[0105] Step S2.3: Select a calibration plate of appropriate size and known parameters, place it in the overlapping field of view of the camera and lidar, and collect point cloud and image data.
[0106] Step S2.4: Use the createSSIM function in OpenCV to create a SSIM calculator, and use the SSIM calculator to calculate the structural similarity index ssim_value between two adjacent frame images.
[0107] Step S2.5: Select the first frame image as the initial key frame, set i=1, j=2, and calculate the structural similarity index ssim_value=SSIMm(〖img〗_i,〖img〗_j), where i, j represent the frame image data.
[0108] Step S2.6: If ssim_value is greater than the threshold value T_ssim, then [img]_i is regarded as a key frame and retained, and i=j, j=j+1. If ssim_value is less than the threshold value T_ssim, then [img]_i and the corresponding point cloud data are removed, and i=i, j=j+1. The threshold value T_ssim is assigned based on experience.
[0109] Step S2.7: Determine whether j is less than N (N represents the total number of image data frames). If so, return to execute step S2.6; otherwise, return to the final retained key frame.
[0110] In this method, by synchronizing the time of the data collected by the radar and the camera, it is easy to select the key data frame, which is conducive to the subsequent use of the image data of the key data frame for camera calibration and the use of the point cloud data of the key data frame for radar calibration. By selecting the key data frame, the data of the non-key data frame is filtered out, which further improves the accuracy of the data and is conducive to improving the calibration efficiency and calibration accuracy.
[0111] Step S403, calibrating the intrinsic parameters of the camera based on the position of the calibration plate in the image data of the key data frame to obtain the camera coordinate system.
[0112] Specifically, the above step S403 includes:
[0113] Step S4031, obtaining a first conversion relationship between the camera's intrinsic coordinate system and the world coordinate system based on the pixel coordinates of the calibration plate in the image data of the key data frame and the world coordinates of the calibration plate.
[0114] Step S4032, obtaining a homography matrix constructed by multiplying the intrinsic parameters and extrinsic parameters of the camera.
[0115] Step S4033, jointly solving the homography matrix and the first conversion relationship to obtain a second conversion relationship between the homography matrix and the intrinsic parameters and extrinsic parameters of the camera.
[0116] Step S4034, construct a symmetric matrix using the homography matrix, solve the symmetric matrix, and obtain the intrinsic parameters of the camera.
[0117] Step S4035, based on the intrinsic parameters of the camera and the second transformation relationship, the extrinsic parameters of the camera are obtained by solving, and the camera coordinate system is determined based on the intrinsic parameters and extrinsic parameters of the camera.
[0118] In one example, the intrinsic parameters of the camera are calibrated according to the acquired chessboard image data, and the specific steps include: S3.1: The chessboard search in the acquired multiple image data is completed through the findChessboardCorners function in OpenCV, including: 1) Reading the input image: reading the chessboard image and storing it as a Mat object. The chessboard is composed of black and white squares; 2) Setting the chessboard size: passing the number of rows and columns of the chessboard as parameters to the findChessboardCorners function; 3) Calling the findChessboardCorners function; 4) Processing the return value: the findChessboardCorners function returns a non-zero value (i.e. true), which means that all corner points have been successfully detected and they are arranged in a certain order (usually from left to right in rows).
[0119] Get the sub-pixel coordinates of the checkerboard corners through the find4QuadCornerSubpix function in OpenCV to find the sub-pixel corners of the quadrilateral. It is used to find the exact corner positions on the checkerboard image. Find the corners of the quadrilateral in the input image and return the sub-pixel accurate positions of the corners. It iteratively approximates the lines that separate the checkerboard field (quadrilateral) and returns the intersections of these lines, thereby achieving sub-pixel positioning of the corners.
[0120] The conversion equation of the first conversion relationship between the camera and the world coordinate system is:
[0121]
[0122] Where (u, v) is the pixel coordinate of the calibration plate in the acquired image data; (X W ,Y W ,Z W ) is the world coordinate of the calibration plate in the acquired image data. Since the world coordinate system is set up on the calibration plate and the Z axis is perpendicular to the calibration plate, Z = 0; A is the intrinsic parameter matrix of the camera, Where γ is the parameter of the inclination angle between the two coordinate axes. If the two coordinate axes are perpendicular to each other, γ = 0. The default value of γ is 0. x ,c y is the camera optical center coordinate, f x , f y is the equivalent focal length; [R t] is the camera extrinsic parameter matrix, specifically, R is a rotation matrix of size 3×3, (r1, r2, r3) are three rotation vectors in the rotation matrix, t is a translation vector of size 3×1; S is a scale factor, which is not limited in the present invention.
[0123] S3.2: Solving the camera internal and external parameter matrix:
[0124] The homography matrix H is used to represent the product of the internal parameters and the external parameters. H is a 3×3 matrix, usually called the homography matrix, written as follows:
[0125]
[0126] Among them, h1 to h9 are the parameter values of rotation and translation respectively.
[0127] Combining the above formula with formula (1), we can get:
[0128]
[0129] Written in matrix form, the formula is:
[0130]
[0131] In formula (4), h is the matrix element of the homography matrix H expanded into a 9×1 vector form, that is:
[0132] h=[h1h2h3h4h5h6h7h8h9] T #(5)
[0133] From formula (4), we can know that the corresponding points of the world coordinates and pixel coordinates of the calibration plate in the acquired image data can construct two equations. Since the homography matrix H has one element as a homogeneous coordinate, the homography matrix H has 8 degrees of freedom. Therefore, the four corresponding points can establish a system of equations and solve the homography matrix H. Because the homography matrix H is the product of the matrix constructed by the intrinsic parameter matrix and the extrinsic parameter, after the homography matrix H is solved, it can be used to further solve the camera's internal and external parameters. The second conversion relationship between the homography matrix H and the camera's internal and external parameters can be expressed by the following formula:
[0134] H=A[r1r2t]=[h1h2h3]#(6)
[0135] When the number of extracted corner points is greater than 4, the least square method is used to solve the homography matrix H. Since there is a unit orthogonal relationship between the column vectors of the extrinsic parameter matrix, that is:
[0136] ‖r1‖=‖r2‖=1#(7)
[0137]
[0138] Here, T is used to represent the transpose of the matrix.
[0139] Substituting into formula (6) we can get:
[0140]
[0141] Each homography matrix can list two equations. Since the intrinsic matrix contains 5 parameters, at least 3 homography matrices are needed to solve. By changing the relative position between the camera and the chessboard, at least 3 chessboard pictures with different postures can be obtained to determine the 3 homography matrices to construct 6 equations. Define the following formula:
[0142]
[0143] Where B is a symmetric matrix. By using formula (9), at least three chessboard images with different postures can be used to obtain the internal parameter matrix A through matrix transformation. The internal parameter elements are solved as follows:
[0144]
[0145] Among them, γ is the parameter of the inclination angle between the two coordinate axes, c x ,c y is the camera optical center coordinate, f x , f y is the equivalent focal length, B 11 , B 12 , B 13 , B 22 , B 23 are all parameters in matrix B. Since B is a symmetric matrix, B 12 =B 21 , B 13 =B 31 , B 23 =B 32 .
[0146] After obtaining the internal parameters, it is easy to solve the external parameters according to formula (6) as follows:
[0147]
[0148] r3=r1×r2#(19)
[0149] Here, λ is an arbitrary scalar.
[0150] In this method, the image data of the calibration plate acquired by the camera and the coordinates of the calibration plate in the world coordinate system are used to solve the intrinsic and extrinsic parameters of the camera, which together constitute the camera coordinate system, thereby facilitating the joint calibration of the camera coordinate system and the radar coordinate system.
[0151] Step S404: Calibrate the external parameters of the radar based on the point cloud data corresponding to the calibration object in the key data frame to obtain the radar coordinate system. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.
[0152] Step S405: perform joint calibration of the camera and the radar based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system to obtain the conversion relationship between the camera coordinate system and the radar coordinate system. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.
[0153] The radar and camera joint calibration method provided in this embodiment facilitates the selection of key data frames by time synchronizing the data collected by the radar and the camera, which is beneficial for the subsequent use of the image data of the key data frames for camera calibration and the use of the point cloud data of the key data frames for radar calibration. By selecting the key data frames, the data of the non-key data frames are filtered out, which further improves the accuracy of the data and helps to improve the calibration efficiency and calibration accuracy. By using the image data of the calibration plate collected by the camera and the coordinates of the calibration plate in the world coordinate system, the intrinsic and extrinsic parameters of the camera are solved to form a camera coordinate system, which facilitates the joint calibration of the camera coordinate system and the radar coordinate system.
[0154] In this embodiment, a radar and camera joint calibration method is provided, which can be used for the above-mentioned computer device. Figure 5 FIG. 4 is a flowchart of another radar and camera joint calibration method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:
[0155] Step S501: adjust the relative positions of the radars and cameras to form a common field of view between the radars and cameras. Figure 4 Step S401 of the illustrated embodiment will not be described in detail here.
[0156] Step S502: Set calibration objects and calibration plates at target positions within the public field of view, and collect image data and point cloud data of key data frames of the calibration objects and calibration plates at different target positions. Figure 4 Step S402 of the illustrated embodiment will not be described in detail here.
[0157] Step S503: Calibrate the camera's internal parameters based on the position of the calibration plate in the image data of the key data frame to obtain the camera coordinate system. Figure 4 Step S403 of the illustrated embodiment will not be described in detail here.
[0158] Step S504, based on the point cloud data corresponding to the calibration object in the key data frame, calibrate the external parameters of the radar to obtain the radar coordinate system.
[0159] Specifically, the above step S504 includes:
[0160] Step S5041, extracting the calibration object point cloud from the point cloud data of the key data frame.
[0161] In an example, this step is explained in detail.
[0162] Step S5042: Set the main radar, use the radar coordinate system corresponding to the main radar as the target coordinate system, and set the initial radar extrinsics between the remaining radars and the main radar.
[0163] Step S5043: Based on the initial radar external parameters, the first point set of the calibration object point cloud in the main radar is registered with the second point set of the calibration object point cloud in the remaining radars to obtain a transformed point set.
[0164] Step S5044: construct an objective function based on the transformation point set, solve the objective function, and obtain a third conversion relationship.
[0165] Step S5045: perform coordinate transformation on the transformation point set based on the third transformation relationship to obtain a second transformation point set.
[0166] Step S5046: based on the second transformation point set, the objective function is iteratively solved to obtain the radar coordinate system.
[0167] In one example, the ICP point cloud registration algorithm is used to calibrate each laser radar external parameter according to the collected multiple sets of calibration object data frames. The specific steps include:
[0168] S4.1: Preprocess the original data frame and extract the calibration point cloud. Specifically, it includes using the RANSAC algorithm to extract and remove the ground point cloud, including using the Euclidean clustering algorithm to remove outliers. Specifically, it includes setting thresholds in the x and y axis directions to remove the background point cloud.
[0169] Specifically, the point cloud is removed by using a model fitting method: the road surface is regarded as a plane, and a road surface model is fitted from the point cloud using methods such as RANSAC, and then the model is used to filter road surface points.
[0170] 2) Removal of outliers:
[0171] In the present invention, the Euclidean clustering algorithm is used to remove outliers.
[0172] 3) Removal of background point cloud:
[0173] The background point cloud is removed by clustering the target point cloud: the non-ground point cloud is clustered into point clusters, where a point cluster represents all the points on a target detected by the laser scan. In this way, the background point cloud can be distinguished from the target point cloud.
[0174] S4.2: Set the main laser radar and determine the main laser radar coordinate system as the target coordinate system, and convert other laser radar coordinate systems to the target coordinate system.
[0175] S4.3: Set the initial external parameters between each laser radar and the main laser radar. Specifically, estimate the initial external parameters T between each laser radar through measurement and mechanical structure parameters. i .
[0176] Specifically, the external parameters of the radar include translation and rotation: 1) The initial external parameter translation (x / y / z), i.e., the distance offset from the target coordinate system in the X / Y / Z axis, is obtained by measuring the actual installation position; 2) The initial value of the initial external parameter rotation (yaw / pitch / roll) parameter is estimated by the rotation angle of the sub-coordinate system coordinate axis to the target coordinate system when the radar is installed horizontally. The initial external parameters are used for initial calibration between radars, and external parameter calibration optimization is performed based on the initial parameters, because the actual installation position is not the ideal installation position and there are measurement errors.
[0177] S4.4, for the point cloud of the calibration object in the main laser radar, the point set P = {p1, p2, ..., p j}, the point set Q in the i-th lidar i = {q i,1 ,q i,2 ,…,q i,j}, according to the initial external parameter T i Complete the rough registration and get the transformation point set Q i ′ .
[0178] Step S4.5: Point set Q i ′ Every point q in i ′ ,j Find the distance q in the point set P i ′ The nearest point p j , forming corresponding point pairs and constructing nonlinear objective function Where N is the point set Q i ′ The number of midpoints is used to solve the optimal transformation objective function (R, t) by least squares.
[0179] S4.6: Use the transformation relationship obtained in step S4.5 to transform the point set Q i ′ The coordinates of the points in are transformed to obtain a new point set Q i ′ ′, calculate the minimum value of the objective function (R, t).
[0180] S4.7: Set the conditions for iteration termination. Specifically, the iteration is stopped when the minimum value of the objective function is less than the threshold; specifically, the iteration is stopped when the transformation amount of the transformation (R, t) matrix calculated by the two iterations is less than the threshold; specifically, the iteration is stopped when the maximum number of iterations is reached. If the iteration termination condition is not met, return to step S4.5 to continue execution.
[0181] In this method, by setting the initial radar extrinsic parameters of the main radar and other radars to the main radar, the transformation point set is obtained by alignment, and the objective function is constructed. After the transformation point set is transformed, the objective function is iteratively solved using the transformed transformation point set to obtain the radar coordinate system, which is convenient for realizing the joint calibration of the camera coordinate system and the radar coordinate system.
[0182] Step S505 , based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system, perform joint calibration of the camera and the radar to obtain a conversion relationship between the camera coordinate system and the radar coordinate system.
[0183] Specifically, the above step S505 includes:
[0184] Step S5051, based on the coordinates of the calibration object in the camera coordinate system, obtain the normal vector of the calibration object in the camera coordinate system.
[0185] Step S5052: Establish constraints on the normal vector of the calibration object in the camera coordinate system to obtain a first error function.
[0186] Step S5053: Based on the coordinates of the calibration object in the radar coordinate system and the first error function, a constraint relationship between the camera coordinate system and the radar coordinate system is established, and a conversion relationship between the camera coordinate system and the radar coordinate system is determined.
[0187] In one example, the camera intrinsic parameters obtained in the above embodiment are obtained, and the camera and the laser radar are jointly calibrated according to the coordinate representation of the checkerboard plane in the camera coordinate system and the laser radar coordinate system. The specific steps include:
[0188] S5.1: Perform corner point detection on the chessboard in the collected image data to obtain the pixel coordinates of the corner points, and solve the rotation transformation external parameters [r 1_i r 2_i r 3_i ] and the translation transformation extrinsic parameter t_i.
[0189] S5.2: Derive the representation of the checkerboard plane in the camera coordinate system: Specifically, the normal vector of the checkerboard plane in the world coordinate system is [0 0 1] T , the normal vector of the chessboard plane in the camera coordinate system is: n c = r3. Among them, n cis the normal vector of the chessboard, and r3 is the third column of the rotation matrix.
[0190] Specifically, the chessboard plane is expressed in the camera coordinate system as: r3·(P c +t WC )=0. Among them, t WC is the translation vector of the world coordinates relative to the camera coordinates. Furthermore, the normal vector of the chessboard plane can be expressed in the camera coordinate system as: N c =(r3·t WC )r3.
[0191] S5.3: Establish constraints on the normal vector of the checkerboard plane in the camera coordinate system. Figure 6 is a schematic diagram of a camera and laser radar joint calibration according to an embodiment of the present invention, such as Figure 6 As shown, the following geometric relationship exists in the camera coordinate system: connect the origin of the camera coordinate system to any point P on the chessboard plane in the camera coordinate system c , the vector P corresponding to this point C The projection on the normal vector is equal to the distance from the plane to the origin. Therefore, for n sets of calibration data under different viewing angles and m calibration targets, the following error function can be established:
[0192]
[0193] S5.4: Establish constraints on camera and lidar extrinsic parameters. Figure 7 is a schematic diagram of a checkerboard plane normal vector constraint in a camera coordinate system according to an embodiment of the present invention, such as Figure 7 As shown, is a set of chessboard point clouds, where m is the number of point clouds, R LC and t LC is the rotation matrix and translation vector between the laser radar coordinate system and the camera coordinate system. For any point on the chessboard in the laser radar coordinate system And the corresponding point in the corresponding camera coordinate system The relationship exists Combined with the error function in S5.3, we get the following error function:
[0194]
[0195] The external parameters between the laser radar coordinate system and the camera coordinate system can be obtained by minimizing the above error function. By minimizing the projection error, the linear optimization problem can be solved to obtain the rotation matrix R LC and the translation vector t LC , which is the external parameter obtained after radar-camera calibration.
[0196] In this way, by using multiple sensors such as radar and camera to simultaneously collect calibration data in the same calibration scene and completing the calibration based on feature extraction based on the point cloud of the calibration plate, the efficiency and accuracy of external parameter calibration between multiple sensors can be improved, the calibration process can be simplified, and it helps to meet the needs of large-scale mass production, and effectively combine multiple laser radars and cameras while ensuring accuracy.
[0197] The radar and camera joint calibration method provided in this embodiment sets the initial radar external parameters of the main radar and other radars to the main radar, aligns to obtain the transformation point set, constructs the target function, transforms the transformation point set, and then uses the transformed transformation point set to iteratively solve the target function to obtain the radar coordinate system, which is convenient for realizing the joint calibration of the camera coordinate system and the radar coordinate system. By using multiple sensors of radar and camera to simultaneously collect calibration data in the same calibration scene and complete the calibration based on the feature extraction of the calibration board point cloud, the external parameter calibration efficiency and calibration accuracy between multiple sensors can be improved, the calibration process is simplified, and it helps to meet the needs of large-scale mass production, and effectively combine multiple laser radars and cameras while ensuring accuracy.
[0198] In this embodiment, a radar and camera joint calibration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0199] This embodiment provides a radar and camera joint calibration device, such as Figure 8 As shown, including:
[0200] The common view area forming module 801 is used to adjust the relative positions of the radars and the cameras to form a common view area between the radars and the cameras. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.
[0201] The data acquisition module 802 is used to set calibration objects at target positions in the public field of view and collect image data and point cloud data of key data frames of the calibration objects at different target positions. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.
[0202] The camera coordinate system calibration module 803 is used to calibrate the camera's internal parameters based on the image data of the key data frame to obtain the camera coordinate system. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.
[0203] The radar coordinate system calibration module 804 is used to calibrate the external parameters of the radar based on the point cloud data of the key data frame to obtain the radar coordinate system. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.
[0204] The joint calibration module 805 is used to perform joint calibration of the camera and the radar based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system, and obtain the conversion relationship between the camera coordinate system and the radar coordinate system. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.
[0205] In some optional implementations, the data acquisition module 802 includes:
[0206] The data acquisition unit is used to acquire point cloud data corresponding to the calibration object at different target positions by using radar, and to acquire image data corresponding to the calibration plate at different target positions by using camera.
[0207] The timestamp data acquisition unit is used to acquire radar timestamp data of the radar and camera timestamp data of the camera.
[0208] The time difference calculation unit is used to match the radar timestamp data with the camera timestamp data on the time axis, and calculate the time difference between the radar timestamp data and the camera timestamp data.
[0209] The time synchronization unit is used to synchronize the radar and the camera based on the time difference, and match the point cloud data and image data at the same time.
[0210] The key frame screening unit is used to screen the key data frame based on the point cloud data and image data at the same time, and determine the image data and point cloud data of the key data frame.
[0211] In some optional implementations, the key frame screening unit includes:
[0212] The structural similarity calculation subunit is used to select the image data of the first data frame as the initial key frame, and calculate the structural similarity between the image data of each data frame and the image data of the adjacent data frames in sequence starting from the initial key frame.
[0213] The structural similarity judgment subunit is used to judge whether the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is greater than a preset threshold.
[0214] The key frame determination subunit is used to determine that the current data frame is a key data frame when the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is greater than a preset threshold.
[0215] The point cloud data removal subunit is used to remove the image data of the current frame and the corresponding point cloud data when the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is less than a preset threshold.
[0216] The data frame traversal subunit is used to traverse until the calculation of the data frames corresponding to all image data is completed.
[0217] In some optional implementations, the camera coordinate system calibration module 803 includes:
[0218] The first conversion relationship determining unit is used to obtain a first conversion relationship between the camera's internal coordinate system and the world coordinate system based on the pixel coordinates of the calibration plate in the image data of the key data frame and the world coordinates of the calibration plate.
[0219] The homography matrix acquisition unit is used to obtain the homography matrix constructed by multiplying the intrinsic parameters and extrinsic parameters of the camera.
[0220] The second conversion relationship determination unit is used to jointly solve the homography matrix and the first conversion relationship to obtain a second conversion relationship between the homography matrix and the intrinsic parameters and extrinsic parameters of the camera.
[0221] The camera intrinsic parameter solving unit is used to construct a symmetric matrix using the homography matrix, solve the symmetric matrix, and obtain the intrinsic parameters of the camera.
[0222] The camera extrinsic parameter solving unit is used to solve the camera's extrinsic parameters based on the camera's intrinsic parameters and the second transformation relationship, and determine the camera coordinate system based on the camera's intrinsic parameters and extrinsic parameters.
[0223] In some optional implementations, the radar coordinate system calibration module 804 includes:
[0224] The calibration object point cloud extraction unit is used to extract the calibration object point cloud from the point cloud data of the key data frame.
[0225] The main radar setting unit is used to set the main radar, use the radar coordinate system corresponding to the main radar as the target coordinate system, and set the initial radar external parameters between the remaining radars and the main radar.
[0226] The transformation point set registration unit is used to register the first point set of the calibration object point cloud in the main radar and the second point set of the calibration object point cloud in the other radars based on the initial radar external parameters to obtain the transformation point set.
[0227] The third conversion relationship determination unit is used to construct an objective function based on the transformation point set, solve the objective function, and obtain a third conversion relationship.
[0228] The coordinate conversion unit is used to perform coordinate conversion on the transformation point set based on the third conversion relationship to obtain a second transformation point set.
[0229] The radar coordinate system solving unit is used to iteratively solve the objective function based on the second transformation point set to obtain the radar coordinate system.
[0230] In some optional implementations, the joint calibration module 805 includes:
[0231] The normal vector determination unit is used to obtain the normal vector of the calibration object in the camera coordinate system based on the coordinates of the calibration object in the camera coordinate system.
[0232] The normal vector constraint unit is used to establish the constraint of the calibration object on the normal vector in the camera coordinate system to obtain the first error function.
[0233] The camera-radar conversion relationship determination unit is used to establish a constraint relationship between the camera coordinate system and the radar coordinate system based on the coordinates of the calibration object in the radar coordinate system and the first error function, and determine the conversion relationship between the camera coordinate system and the radar coordinate system.
[0234] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.
[0235] The radar and camera joint calibration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.
[0236] The embodiment of the present invention also provides a computer device having the above Figure 8 The radar and camera joint calibration device shown.
[0237] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.
[0238] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.
[0239] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.
[0240] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0241] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.
[0242] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.
[0243] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.
[0244] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.
[0245] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.
[0246] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.
Claims
1. A radar and camera joint calibration method, characterized in that: Applied to a joint system consisting of several radars and cameras, the method comprises: Adjusting the relative posture of each of the radars and the camera to form a common field of view area between the radar and the camera; Setting a calibration object and a calibration plate at a target position within the public field of view, and collecting image data and point cloud data of key data frames of the calibration object and the calibration plate at different target positions; Calibrate the intrinsic parameters of the camera based on the position of the calibration plate in the image data of the key data frame to obtain a camera coordinate system; Based on the point cloud data corresponding to the calibration object in the key data frame, calibrate the external parameters of the radar to obtain a radar coordinate system; Based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system, the camera and the radar are jointly calibrated to obtain a conversion relationship between the camera coordinate system and the radar coordinate system.
2. The method according to claim 1, characterized in that The collecting of image data and point cloud data of key data frames of the calibration object and the calibration plate at different target positions includes: The radar is used to acquire point cloud data corresponding to the calibration object at different target positions, and the camera is used to acquire image data corresponding to the calibration plate at different target positions; Collecting radar timestamp data of the radar and camera timestamp data of the camera; Matching the radar timestamp data with the camera timestamp data on a time axis, and calculating a time difference between the radar timestamp data and the camera timestamp data; Based on the time difference, time synchronization is performed on the radar and the camera to match and obtain the point cloud data and the image data at the same time; Based on the point cloud data and the image data at the same time, the key data frame is screened and obtained, and the image data and the point cloud data of the key data frame are determined.
3. The method according to claim 2, characterized in that The step of screening the key data frame based on the point cloud data and the image data at the same time includes: Selecting the image data of the first data frame as an initial key frame, and calculating the structural similarity between the image data of each data frame and the image data of adjacent data frames in sequence starting from the initial key frame; Determine whether the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is greater than a preset threshold; When the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is greater than a preset threshold, determining the current data frame as a key data frame; When the structural similarity between the image data of the current data frame and the image data of the adjacent data frame is less than a preset threshold, the image data of the current frame and the corresponding point cloud data are removed; Traverse until the data frames corresponding to all image data are calculated.
4. The method according to claim 1, characterized in that: The step of calibrating the intrinsic parameters of the camera based on the image data of the key data frame to obtain a camera coordinate system includes: Obtaining a first conversion relationship between an internal coordinate system of the camera and a world coordinate system based on pixel coordinates of the calibration plate in the image data of the key data frame and world coordinates of the calibration plate; Obtaining a homography matrix constructed by multiplying the intrinsic parameters and extrinsic parameters of the camera; The homography matrix and the first conversion relationship are solved jointly to obtain a second conversion relationship between the homography matrix and the intrinsic parameters and extrinsic parameters of the camera; Using the homography matrix to construct a symmetric matrix, solving the symmetric matrix to obtain the intrinsic parameters of the camera; Based on the intrinsic parameters of the camera and the second transformation relationship, the extrinsic parameters of the camera are obtained by solving, and the camera coordinate system is determined based on the intrinsic parameters and extrinsic parameters of the camera.
5. The method according to claim 1, characterized in that The step of calibrating the external parameters of the radar based on the point cloud data of the key data frame to obtain a radar coordinate system includes: Extracting the calibration object point cloud from the point cloud data of the key data frame; Setting a main radar, taking the radar coordinate system corresponding to the main radar as the target coordinate system, and setting initial radar extrinsics between the remaining radars and the main radar; Based on the initial radar extrinsic parameters, registering a first point set of the calibration object point cloud in the primary radar with a second point set of the calibration object point cloud in the remaining radars to obtain a transformed point set; Based on the transformation point set, construct an objective function, solve the objective function, and obtain a third conversion relationship; Performing coordinate transformation on the transformation point set based on the third transformation relationship to obtain a second transformation point set; The objective function is iteratively solved based on the second transformation point set to obtain the radar coordinate system.
6. The method according to claim 1, characterized in that The joint calibration of the camera and the radar based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system to obtain a conversion relationship between the camera coordinate system and the radar coordinate system includes: Based on the coordinates of the calibration object in the camera coordinate system, obtaining a normal vector of the calibration object in the camera coordinate system; Establishing a constraint of the calibration object on the normal vector in the camera coordinate system to obtain a first error function; Based on the coordinates of the calibration object in the radar coordinate system and the first error function, a constraint relationship between the camera coordinate system and the radar coordinate system is established, and a conversion relationship between the camera coordinate system and the radar coordinate system is determined.
7. A radar and camera joint calibration device, characterized in that: The device comprises: A common field of view area forming module, used for adjusting the relative posture of each of the radars and the camera to form a common field of view area between the radars and the camera; A data acquisition module, used to set calibration objects at target positions within the public field of view, and to collect image data and point cloud data of key data frames of the calibration objects at different target positions; A camera coordinate system calibration module, used to calibrate the intrinsic parameters of the camera based on the image data of the key data frame to obtain a camera coordinate system; A radar coordinate system calibration module, used to calibrate the external parameters of the radar based on the point cloud data of the key data frame to obtain a radar coordinate system; The joint calibration module is used to perform joint calibration of the camera and the radar based on the coordinates of the calibration object in the camera coordinate system and the radar coordinate system to obtain a conversion relationship between the camera coordinate system and the radar coordinate system.
8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the radar and camera joint calibration method according to any one of claims 1 to 6 by executing the computer instructions.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the radar and camera joint calibration method according to any one of claims 1 to 6.
10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the radar and camera joint calibration method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Sensor calibration method and device, storage medium and calibration system
CN112819896A
Multi-laser radar and multi-camera combined calibration method and system
CN113592957A
Joint calibration method for laser radar and camera
CN114140534A
Radar and camera joint calibration method and device, electronic equipment and storage medium
CN115359130A
Calibration method and device between laser radars based on entity calibration
CN115840196A
Cited By
Sensor external parameter calibration method and device
CN121616671A
