Joint calibration method and system for laser radar and camera, vehicle and equipment
By using a combined calibration method of camera and lidar, and utilizing marker images and point cloud data, the calibration challenge of vehicle-mounted lidar after changes in extrinsic parameters was solved, achieving high-precision calibration in non-factory environments.
Patent Information
- Application Number
- CN202511099489.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-06
- Publication Date
- 2025-12-16
AI Technical Summary
In existing technologies, it is difficult to recalibrate vehicle-mounted LiDAR efficiently and accurately in non-factory environments after changes in extrinsic parameters, resulting in inaccurate perception results.
By using a joint calibration method combining camera and LiDAR, the outline and point cloud of the marker are extracted using the image of the marker captured by the camera and the point cloud data of the LiDAR, and the extrinsic parameters of the LiDAR are solved, thus enabling a convenient calibration operation.
The joint calibration of cameras and lidar can be completed in the daily driving environment of vehicles. It is simple to operate, highly implementable, and has high calibration accuracy.
Smart Images
Figure CN121147321A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle technology, and in particular to a joint calibration method, system, vehicle, and device for lidar and camera. Background Technology
[0002] For automotive LiDAR systems, after a period of use or after-sales maintenance, the extrinsic parameters of the LiDAR will change, requiring recalibration to ensure the accuracy of the LiDAR's perception results. Generally, the factory performs calibration using a dedicated LiDAR target when the vehicle leaves the production line. However, for the scenarios described above, vehicles cannot be returned to the factory every time, thus limiting its applicability.
[0003] Among related technologies, there is a method of obtaining accurate extrinsic parameters of LiDAR through joint calibration of LiDAR and camera. Currently, commonly used joint calibration schemes for LiDAR targets and cameras are as follows: calibration is performed using a customized target that can be accurately detected by both LiDAR and camera; calibration is performed by matching point and line features extracted from camera images and LiDAR point clouds; and joint calibration is performed based on a deep learning model. However, the following drawbacks exist:
[0004] Calibration can be achieved by using customized targets that can be accurately detected by both LiDAR and cameras. However, the production of these targets is time-consuming and labor-intensive, and they are generally only suitable for vehicle calibration in factories. Calibration can also be achieved by extracting point and line features from camera images and LiDAR point clouds for matching. However, the extraction and matching of point and line features are difficult and prone to errors in extraction and calibration. Therefore, this method is mainly suitable for manual calibration and cannot achieve automatic calibration well. Joint calibration based on deep learning models has limited data and is difficult to train, resulting in poor calibration accuracy. Summary of the Invention
[0005] Therefore, it is necessary to provide a joint calibration method, system, vehicle, and equipment for LiDAR and camera to address the above-mentioned technical problems. This method can perform joint calibration of camera and LiDAR in the daily driving environment of the vehicle or in a simple setting. It is not only convenient to calibrate, with simple scenarios and high feasibility, but also has the advantage of high calibration accuracy of LiDAR.
[0006] Firstly, a joint calibration method for lidar and camera is provided, including:
[0007] Obtain images of the landmark captured by the camera;
[0008] Based on the image of the marker, the outline of the marker is extracted;
[0009] Obtain point cloud scan data of the marker scanned by a lidar, wherein the lidar has initial lidar extrinsic parameters;
[0010] The point cloud of the marker is extracted based on the point cloud scan data of the marker;
[0011] The point cloud of the marker is converted onto an image to obtain a point cloud image, and the extrinsic parameters of the lidar are solved based on the positional relationship between the points on the point cloud image and the outline of the marker.
[0012] In some examples, extracting the outline of the marker from its image includes:
[0013] Based on the camera's intrinsic parameter matrix and distortion model, the 2D points corresponding to the 3D points of the marker are obtained, resulting in a 2D image.
[0014] The pixels on the 2D image are augmented to obtain an augmented image;
[0015] The outline of the marker is extracted from the augmented image.
[0016] In some examples, extracting the point cloud of the marker based on the point cloud scan data of the marker includes:
[0017] The point cloud scan data of the markers is expanded to obtain expanded point cloud scan data;
[0018] Based on the information of the marker, the point cloud augmentation scan data is filtered to obtain the point cloud of the marker.
[0019] In some examples, the process of converting the point cloud of the marker onto an image to obtain a point cloud image, and then solving for the extrinsic parameters of the lidar based on the positional relationship between the points in the point cloud image and the contour of the marker, includes:
[0020] The point cloud of the marker is converted onto an image to obtain the point cloud image;
[0021] Determine whether a point on the point cloud image is located within the outline of the landmark;
[0022] If not, then the positional error between the point and the edge of the marker's outline is obtained;
[0023] Based on the position error, the extrinsic parameters of the lidar are determined.
[0024] In some examples, solving for the extrinsic parameters of the lidar based on the position error includes:
[0025] Define the optimization objective;
[0026] Based on the optimization objective, the positional error between each point on the point cloud image and the edge of the outline of the marker is optimized by error accumulation, and the external parameters of the camera and lidar with the smallest error are solved.
[0027] In some examples, after solving for the extrinsic parameters of the lidar, the process further includes calibrating the lidar based on the solved extrinsic parameters.
[0028] In some examples, before extracting the outline of the marker from the image of the marker, the process further includes:
[0029] Convert the image of the marker into a black and white image.
[0030] Secondly, a joint calibration system for lidar and camera is provided, comprising:
[0031] The acquisition module is used to acquire images of the marker captured by a camera and point cloud scan data of the marker obtained by a lidar scanner, wherein the lidar has initial lidar extrinsic parameters.
[0032] The extraction module is used to extract the outline of the marker based on the image of the marker, and to extract the point cloud of the marker based on the point cloud scan data of the marker;
[0033] The solution module is used to convert the point cloud of the marker onto an image to obtain a point cloud image, and to solve for the extrinsic parameters of the lidar based on the positional relationship between the points on the point cloud image and the contour of the marker.
[0034] Thirdly, a vehicle is provided, comprising: a joint calibration system of lidar and camera as described in the second aspect above.
[0035] Fourthly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the joint calibration method for a lidar and a camera as described in the first aspect and any possible implementation of the first aspect.
[0036] Fifthly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the joint calibration method for a lidar and a camera as described in the first aspect and any possible implementation thereof.
[0037] In a sixth aspect, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the joint calibration method for lidar and camera described in the first aspect and any possible implementation thereof.
[0038] Using the embodiments of this application, an image of a marker captured by a camera and point cloud scan data of the marker scanned by a LiDAR with initial extrinsic parameters are obtained. Then, the outline of the marker is extracted from the image, and the point cloud of the marker is extracted from the point cloud scan data. Finally, the point cloud of the marker is converted onto the image to obtain a point cloud image, and the extrinsic parameters of the LiDAR are solved based on the positional relationship between the points on the point cloud image and the outline of the marker. Joint calibration of the camera and LiDAR can be performed in the daily driving environment of a vehicle or in a simply arranged environment. This not only offers convenient calibration operations, simple scenarios, and high feasibility, but also has the advantage of high LiDAR calibration accuracy. Attached Figure Description
[0039] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0040] Figure 1 A flowchart illustrating the joint calibration method for lidar and camera provided in this application embodiment;
[0041] Figure 2 A schematic diagram illustrating the joint calibration method for lidar and camera provided in an embodiment of this application;
[0042] Figure 3 A structural block diagram of the joint calibration system for lidar and camera provided in the embodiments of this application;
[0043] Figure 4 A structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation
[0044] The present application will now be described in further detail with reference to the embodiments and accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the application are shown in the accompanying drawings.
[0045] It should be noted that, unless otherwise specified, the embodiments and features of the embodiments in this application can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments.
[0046] The following describes in detail, with reference to the accompanying drawings, a joint calibration method, system, vehicle, and device for lidar and camera according to embodiments of this application.
[0047] Figure 1 This is a flowchart of a joint calibration method for a lidar and a camera according to an embodiment of this application. Figure 1 As shown, and in combination Figure 2 The joint calibration method for lidar and camera according to embodiments of this application includes the following steps:
[0048] S101: Obtain an image of the marker taken by the camera.
[0049] First, create a scene. For example, on a road you drive on, use road signs, lampposts, etc. as landmarks to create a shooting scene. Of course, you can also place landmarks in an open area to create a shooting scene.
[0050] Next, set the approximate location of the marker, move the vehicle to align the marker and the vehicle in the set position, and then take an image of the marker using the onboard camera.
[0051] S102: Extract the outline of the marker based on the image of the marker.
[0052] In one embodiment of this application, extracting the outline of the marker based on the image of the marker includes: obtaining 2D points corresponding to the 3D points of the marker based on the camera's intrinsic parameter matrix and distortion model to obtain a 2D image; expanding the pixels on the 2D image to obtain an expanded image; and extracting the outline of the marker from the expanded image.
[0053] Before extracting the outline of the marker from its image, the method further includes converting the image of the marker into a black and white image.
[0054] Specifically, based on the marker information, the approximate position of the marker on the marker image is obtained, and the 2S point P corresponding to the 3D point P is obtained through the camera intrinsic parameter matrix K and the distortion model D. uv :
[0055]
[0056] Since the marker's location is not precise, the calculated pixels are enlarged to ensure that the marker ultimately falls within the enlarged image area.
[0057]
[0058] p leftupnew =p middle +1.5*(p middle -p leftup )
[0059] p leftdownnew =p middle +1.5*(p middle -p leftdown )
[0060] p rightupnew =p middle +1.5*(p middle -p rightup )
[0061] p rightdownnew =p middle +1.5*(p middle -p rightdown (2)
[0062] Extract the precise location of the marker within the acquired range to obtain the marker's outline, for example:
[0063] First, a clear black-and-white image is obtained by using methods such as median filtering, bilateral filtering, and adaptive image binarization. Then, the ConvexHull algorithm is used to obtain the contour information within the target area. Finally, the approxPoly fitting algorithm is used to fit the obtained contour information into the shape of the marker, thereby obtaining the contour of the marker. If multiple marker contours are obtained, they can be filtered according to parameters such as contour area and shape to obtain the contour information of the target marker.
[0064] S103: Obtain point cloud scanning data of the marker scanned by the lidar, wherein the lidar has initial lidar extrinsic parameters.
[0065] Specifically, based on the marker information, the corresponding point cloud can be extracted using basic point cloud filtering algorithms.
[0066] It should be noted that because the initial extrinsic parameters of lidar have low accuracy, the extraction range needs to be expanded. Generally, this can be achieved by directly using PCL filtering algorithms.
[0067] S104: Extract the point cloud of the marker based on the point cloud scanning data of the marker.
[0068] In one embodiment of this application, extracting the point cloud of the marker based on the point cloud scanning data of the marker includes: expanding the point cloud scanning data of the marker to obtain expanded point cloud scanning data; and filtering the expanded point cloud scanning data based on the information of the marker to obtain the point cloud of the marker.
[0069] S105: Convert the point cloud of the marker onto an image to obtain a point cloud image, and solve for the extrinsic parameters of the lidar based on the positional relationship between the points on the point cloud image and the outline of the marker.
[0070] In one embodiment of this application, the step of converting the point cloud of the marker onto an image to obtain a point cloud image, and solving for the extrinsic parameters of the lidar based on the positional relationship between the points in the point cloud image and the contour of the marker, includes: converting the point cloud of the marker onto an image to obtain the point cloud image; determining whether the points in the point cloud image are located within the contour of the marker; if not, obtaining the positional error between the points and the edge of the contour of the marker; and solving for the extrinsic parameters of the lidar based on the positional error.
[0071] In this example, the extrinsic parameters of the lidar are solved based on the position error, including: determining the optimization target; and based on the optimization target, performing error accumulation optimization on the position error between each point on the point cloud image and the edge of the outline of the marker, and solving for the extrinsic parameters of the camera and lidar with the smallest error.
[0072] Furthermore, after solving for the extrinsic parameters of the lidar, the process also includes calibrating the lidar based on the solved extrinsic parameters.
[0073] Specifically, optimization equations are established using 3D point clouds and 2D landmark outlines to obtain the extrinsic parameters of the LiDAR. For example:
[0074] Convert the 3D point cloud onto the image and confirm whether the point cloud is within the extracted contour.
[0075] If the point is within the contour, the error is 0; otherwise, the error is the distance from the 3D point cloud to the edge of the contour.
[0076]
[0077] The overall optimization goal is as follows: to optimize the point cloud of the landmark to fit within the landmark's 2D outline.
[0078]
[0079] Objective solution:
[0080] Generally, the above equations can be directly filled into the relevant optimization library, and the relevant functions can be called to quickly and efficiently complete the solution and obtain the external parameters of the lidar.
[0081] In one embodiment of this application, the optimized Tcam2lidar can be used to project the point cloud onto the image, and the obtained 2D pixel points can be adjusted according to the point cloud intensity to adjust the corresponding RBG value. Then, visualization can be performed to confirm the final calibration accuracy.
[0082] According to the joint calibration method of LiDAR and camera according to the embodiments of this application, an image of a marker captured by the camera and point cloud scanning data of the marker scanned by the LiDAR with initial extrinsic parameters are obtained. Then, the outline of the marker is extracted based on the image of the marker, and the point cloud of the marker is extracted based on the point cloud scanning data of the marker. Finally, the point cloud of the marker is converted onto the image to obtain a point cloud image, and the extrinsic parameters of the LiDAR are solved based on the positional relationship between the points on the point cloud image and the outline of the marker. The joint calibration of camera and LiDAR can be completed in the daily driving environment of a vehicle or in a simple environment. It is not only convenient to calibrate, with simple scenarios and high feasibility, but also has the advantages of high LiDAR calibration accuracy.
[0083] Figure 3 This is a structural block diagram of a joint calibration system for a lidar and a camera according to an embodiment of this application. Figure 3 As shown, the joint calibration system for lidar and camera according to an embodiment of this application includes: an acquisition module 310, an extraction module 320, and a solution module 330, wherein:
[0084] The acquisition module 310 is used to acquire images of the marker captured by a camera and point cloud scan data of the marker obtained by a lidar scanner, wherein the lidar has initial lidar extrinsic parameters.
[0085] The extraction module 320 is used to extract the outline of the marker based on the image of the marker, and to extract the point cloud of the marker based on the point cloud scan data of the marker;
[0086] The solver module 330 is used to convert the point cloud of the marker onto an image to obtain a point cloud image, and to solve for the extrinsic parameters of the lidar based on the positional relationship between the points on the point cloud image and the outline of the marker.
[0087] The joint calibration system for LiDAR and camera according to embodiments of this application obtains an image of a marker captured by the camera and point cloud scan data of the marker scanned by the LiDAR with initial extrinsic parameters. Then, based on the image of the marker, the outline of the marker is extracted, and based on the point cloud scan data of the marker, the point cloud of the marker is extracted. Finally, the point cloud of the marker is converted onto the image to obtain a point cloud image, and the extrinsic parameters of the LiDAR are solved based on the positional relationship between the points on the point cloud image and the outline of the marker. The joint calibration of the camera and LiDAR can be performed in the daily driving environment of a vehicle or in a simply arranged environment. It not only offers convenient calibration operation, simple scenarios, and high feasibility, but also has the advantage of high LiDAR calibration accuracy.
[0088] Specific limitations regarding the joint calibration system for LiDAR and cameras can be found in the limitations of the joint calibration method for LiDAR and cameras described above, and will not be repeated here. Each module of the aforementioned joint calibration system for LiDAR and cameras can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0089] Furthermore, a vehicle is provided, comprising: a joint calibration system for a lidar and a camera according to any of the above embodiments. After acquiring an image of a marker captured by the camera and point cloud scan data of the marker scanned by the lidar with initial extrinsic parameters, the vehicle can extract the marker's outline from the marker image and extract the marker's point cloud from the point cloud scan data. Finally, the point cloud of the marker is converted onto an image to obtain a point cloud image, and the extrinsic parameters of the lidar are solved based on the positional relationship between the points in the point cloud image and the marker's outline. This system allows for joint calibration of the camera and lidar in the vehicle's daily driving environment or in a simply arranged environment. It not only offers convenient calibration operations, simple scenarios, and high feasibility, but also boasts the advantage of high lidar calibration accuracy.
[0090] Furthermore, other components and functions of the vehicle according to the embodiments of this application are known to those skilled in the art and will not be described in detail here.
[0091] In one embodiment, a computer device is provided. Figure 4 This is a structural block diagram of the computer device provided in the embodiments of this application, with reference to... Figure 4 The computer device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned embodiment of the joint calibration method for the lidar and camera. For example, it executes: obtaining an image of a marker captured by the camera;
[0092] Based on the image of the marker, the outline of the marker is extracted;
[0093] Obtain point cloud scan data of the marker scanned by a lidar, wherein the lidar has initial lidar extrinsic parameters;
[0094] The point cloud of the marker is extracted based on the point cloud scan data of the marker;
[0095] The point cloud of the marker is converted onto an image to obtain a point cloud image, and the extrinsic parameters of the lidar are solved based on the positional relationship between the points on the point cloud image and the outline of the marker.
[0096] This application also provides a computer-readable storage medium storing a computer program. When the processor executes the computer program, it implements the aforementioned embodiment of the joint calibration method for LiDAR and camera. For example, it executes: obtaining an image of a marker captured by the camera;
[0097] Based on the image of the marker, the outline of the marker is extracted;
[0098] Obtain point cloud scan data of the marker scanned by a lidar, wherein the lidar has initial lidar extrinsic parameters;
[0099] The point cloud of the marker is extracted based on the point cloud scan data of the marker;
[0100] The point cloud of the marker is converted onto an image to obtain a point cloud image, and the extrinsic parameters of the lidar are solved based on the positional relationship between the points on the point cloud image and the outline of the marker.
[0101] This application provides a computer program product including instructions that, when executed, cause the method described in this application embodiment to be performed. For example, it can execute... Figure 1 The steps of the joint calibration method of lidar and camera shown include, for example, obtaining an image of the marker captured by the camera;
[0102] Based on the image of the marker, the outline of the marker is extracted;
[0103] Obtain point cloud scan data of the marker scanned by a lidar, wherein the lidar has initial lidar extrinsic parameters;
[0104] The point cloud of the marker is extracted based on the point cloud scan data of the marker;
[0105] The point cloud of the marker is converted onto an image to obtain a point cloud image, and the extrinsic parameters of the lidar are solved based on the positional relationship between the points on the point cloud image and the outline of the marker.
[0106] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0107] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0108] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A joint calibration method for lidar and camera, characterized in that, include: Obtain images of the landmark captured by the camera; Based on the image of the marker, the outline of the marker is extracted; Obtain point cloud scan data of the marker scanned by a lidar, wherein the lidar has initial lidar extrinsic parameters; The point cloud of the marker is extracted based on the point cloud scan data of the marker; The point cloud of the marker is converted onto an image to obtain a point cloud image, and the extrinsic parameters of the lidar are solved based on the positional relationship between the points on the point cloud image and the outline of the marker.
2. The joint calibration method for lidar and camera according to claim 1, characterized in that, The step of extracting the outline of the marker based on the image of the marker includes: Based on the camera's intrinsic parameter matrix and distortion model, the 2D points corresponding to the 3D points of the marker are obtained, resulting in a 2D image. The pixels on the 2D image are augmented to obtain an augmented image; The outline of the marker is extracted from the augmented image.
3. The joint calibration method for lidar and camera according to claim 1, characterized in that, The step of extracting the point cloud of the marker based on the point cloud scanning data of the marker includes: The point cloud scan data of the markers is expanded to obtain expanded point cloud scan data; Based on the information of the marker, the point cloud augmentation scan data is filtered to obtain the point cloud of the marker.
4. The joint calibration method for lidar and camera according to any one of claims 1-3, characterized in that, The process of converting the point cloud of the marker onto an image to obtain a point cloud image, and then solving for the extrinsic parameters of the lidar based on the positional relationship between the points in the point cloud image and the contour of the marker, includes: The point cloud of the marker is converted onto an image to obtain the point cloud image; Determine whether a point on the point cloud image is located within the outline of the landmark; If not, then the positional error between the point and the edge of the marker's outline is obtained; Based on the position error, the extrinsic parameters of the lidar are determined.
5. The joint calibration method for lidar and camera according to claim 4, characterized in that, The step of solving the extrinsic parameters of the lidar based on the position error includes: Define the optimization objective; Based on the optimization objective, the positional error between each point on the point cloud image and the edge of the outline of the marker is optimized by error accumulation, and the external parameters of the camera and lidar with the smallest error are solved.
6. The joint calibration method for lidar and camera according to claim 1, characterized in that, After obtaining the extrinsic parameters of the lidar, the process further includes calibrating the lidar based on the obtained extrinsic parameters.
7. The joint calibration method for lidar and camera according to claim 1, characterized in that, Before extracting the outline of the marker from its image, the method further includes: Convert the image of the marker into a black and white image.
8. A joint calibration system for lidar and camera, characterized in that, include: The acquisition module is used to acquire images of the marker captured by a camera and point cloud scan data of the marker obtained by a lidar scanner, wherein the lidar has initial lidar extrinsic parameters. The extraction module is used to extract the outline of the marker based on the image of the marker, and to extract the point cloud of the marker based on the point cloud scan data of the marker; The solution module is used to convert the point cloud of the marker onto an image to obtain a point cloud image, and to solve for the extrinsic parameters of the lidar based on the positional relationship between the points on the point cloud image and the contour of the marker.
9. A vehicle, characterized in that, include: The joint calibration system for lidar and camera as described in claim 8.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the joint calibration method for the lidar and camera according to any one of claims 1-7.