Method, device and system for joint calibration of camera and laser radar
Through the joint inspection and calibration method of camera and lidar, the internal and external parameters of camera and lidar in drone surveying and mapping are automatically identified and calibrated, which solves the problem of poor internal error handling of lidar in the existing technology, and improves the inspection and calibration efficiency and accuracy , realizes automatic and rapid calibration in the field of drone surveying and mapping.
Patent Information
- Application Number
- CN202410929264.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-11
AI Technical Summary
The inspection and calibration fields of existing drone cameras and lidars are usually carried out independently, failing to effectively handle the internal error of lidar and have low processing efficiency.
Using the joint inspection and calibration method of camera and lidar, the internal and external parameters of the camera and lidar are calibrated by automatically identifying the auxiliary positioning points and targets of the inspection field, and the error correction of the three-dimensional point cloud and reference point cloud of lidar are used to realize the camera. Calibration of relative position relationship with lidar.
It improves the efficiency and accuracy of inspection and calibration, and realizes automated, rapid and accurate joint calibration in the field of drone surveying and mapping.
Smart Images

Figure CN118465741B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of drone technology, and in particular to a method, device and system for joint calibration of a camera and a laser radar. Background Art
[0002] With the development of drones, their aerial photogrammetry technology has been widely used, and the calibration workload of drone cameras and lidars has also increased. The current camera calibration field is usually used to calibrate cameras and lidars independently. When the independent calibration results are fused, the internal errors of the lidar are not processed and the processing efficiency is low. Summary of the invention
[0003] The embodiments of the present application provide a method, device and system for joint calibration of a camera and a laser radar, which can realize a one-time automatic joint calibration of the internal and external parameters of the camera and the laser radar, thereby improving the efficiency and accuracy of the calibration.
[0004] On the one hand, an embodiment of the present application provides a method for joint calibration of a camera and a laser radar, including:
[0005] Automatically identify auxiliary positioning points and target points in an image of a calibration field taken by a camera, and calibrate the internal and external orientation elements of the camera according to the identification results;
[0006] According to the original three-dimensional point cloud acquired by the laser radar and the preset reference point cloud, the internal orientation elements and the external orientation elements of the laser radar are calibrated, and the external orientation elements of the camera and the external orientation elements of the laser radar are the same coordinate system elements;
[0007] According to the relative relationship between the external orientation elements of the camera and the external orientation elements of the laser radar, the relative external orientation elements of the camera and the laser radar are jointly calibrated, and the relative external orientation elements of the camera and the laser radar include the relative position relationship between the camera and the laser radar.
[0008] In one aspect, an embodiment of the present application further provides a device for joint calibration of a camera and a laser radar, comprising:
[0009] A recognition module, used for automatically recognizing auxiliary positioning points and target points in the image of the calibration field taken by the camera;
[0010] A calibration module, used for calibrating the internal orientation elements and the external orientation elements of the camera according to the recognition result;
[0011] The calibration module is further used to calibrate the internal orientation elements and external orientation elements of the laser radar according to the original three-dimensional point cloud obtained by the laser radar and the preset reference point cloud, and the external orientation elements of the camera and the external orientation elements of the laser radar are the same coordinate system elements;
[0012] The calibration module is also used to jointly calibrate the relative external orientation elements of the camera and the laser radar according to the relative relationship between the external orientation elements of the camera and the laser radar. The relative external orientation elements of the camera and the laser radar include the relative position relationship between the camera and the laser radar.
[0013] In one aspect, the embodiment of the present application further provides a system for joint calibration of a camera and a laser radar, comprising: a calibration device, a camera, a laser radar, and a calibration field provided with auxiliary positioning points and target points;
[0014] The calibration device is used to automatically identify the auxiliary positioning point and the target point in the image of the calibration field taken by the camera, and calibrate the internal orientation elements and external orientation elements of the camera according to the recognition result;
[0015] According to the original three-dimensional point cloud acquired by the laser radar and the preset reference point cloud, the internal orientation elements and the external orientation elements of the laser radar are calibrated, and the external orientation elements of the camera and the external orientation elements of the laser radar are the same coordinate system elements;
[0016] The relative external orientation elements of the camera and the laser radar are jointly calibrated according to the relative relationship between the external orientation elements of the camera and the laser radar, and the relative external orientation elements of the camera and the laser radar include the relative position relationship between the camera and the laser radar.
[0017] It can be seen from the above-mentioned embodiments of the present application that the method, device and system for joint calibration of a camera and a laser radar can automatically identify auxiliary positioning points and target points in the image of the calibration field taken by the camera, calibrate the internal orientation elements and external orientation elements of the camera, and calibrate the internal orientation elements and external orientation elements of the laser radar based on the original three-dimensional point cloud obtained by the laser radar and the preset reference point cloud. Compared with the prior art, the internal error of the laser radar is corrected when calibrating the internal orientation elements of the laser radar. The external orientation elements of the camera and the external orientation elements of the laser radar are elements of the same coordinate system. Finally, the relative external orientation elements of the camera and the laser radar are jointly calibrated based on the relative relationship between the external orientation elements of the camera and the external orientation elements of the laser radar. The relative external orientation elements characterize the relative position relationship between the camera and the laser radar, thereby realizing automatic, fast and accurate joint calibration of cameras and laser radars in the field of drone mapping. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.
[0019] Figure 1 A schematic diagram of an application scenario of a method for joint calibration of a camera and a laser radar provided in an embodiment of the present application;
[0020] Figure 2A A schematic diagram of the style of auxiliary positioning points in the method for joint calibration of a camera and a laser radar provided in an embodiment of the present application;
[0021] Figure 2B A schematic diagram of the style of auxiliary positioning points in a method for joint calibration of a camera and a laser radar provided in another embodiment of the present application;
[0022] Figure 3A A schematic diagram of a target point pattern in a method for joint calibration of a camera and a laser radar provided in an embodiment of the present application;
[0023] Figure 3B A schematic diagram of a target point pattern in a method for joint calibration of a camera and a laser radar provided in another embodiment of the present application;
[0024] Figure 4 A schematic diagram of the process of the method for joint calibration of a camera and a laser radar in an embodiment of the present application;
[0025] Figure 5 Schematic diagram of the structure of the device for joint calibration of the camera and laser radar in the embodiment of the present application. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0027] The present application embodiment provides a method for completing the joint internal and external reference calibration of a camera and a laser radar at one time. Figure 1 , Figure 1A schematic diagram of an application scenario of the method for joint calibration of a camera and a laser radar provided in an embodiment of the present application, wherein the calibration device 10 is connected to the camera 20 and the laser radar 30 to be calibrated by wireless or wired means, respectively, and the calibration device 10 may be a computer device, a smart mobile terminal or other electronic device having at least a processor, a storage medium and input and output devices. The calibration device 10 controls the camera 20 to shoot the auxiliary positioning points 50 and the target points 60 in the calibration field 40 to obtain an image including the auxiliary positioning points 50 and the target points 60. The calibration device 10 controls the laser radar 30 to obtain the indoor environment point cloud of the calibration field 40.
[0028] The auxiliary positioning point 50 is set in the calibration field 40 to assist in determining the position of the target point 60. The auxiliary positioning point 50 is a coded point with different patterns of center points, for example, a coded point in the form of Chilitags. The patterns and numbers of the auxiliary positioning points are different. According to the center points of the patterns of different auxiliary positioning points identified from the image, the positions and numbers of the auxiliary positioning points can be obtained. For examples of the auxiliary positioning points 50, see Figure 2A and Figure 2B The style of the auxiliary positioning point 50 in the embodiment of the present application is not affected by Figure 2A and Figure 2B restrictions.
[0029] The target point 60 can be set on the wall, ground and three-dimensional rack of the calibration field 40. The three-dimensional rack can be a column or various forms of three-dimensional bodies, such as a grid frame, a ground column and a top hanging column; the ground target point 60 can be in the form of a bracket or can be directly flat on the ground.
[0030] The inner contour of the target point 60 is a circle with a measurement center, and the outer contour shape is not limited. Figure 3A and Figure 3B The style of the auxiliary positioning point 50 in the embodiment of the present application is not affected by Figure 3A and Figure 3B restrictions.
[0031] See also Figure 4 , Figure 1 A flowchart of a method for joint calibration of a camera and a laser radar provided in an embodiment of the present application is provided. For ease of explanation, only the part related to the embodiment of the present application is shown. The method may include:
[0032] S101, automatically identifying auxiliary positioning points and target points in an image of a calibration field taken by a camera, and calibrating internal and external orientation elements of the camera according to the identification results;
[0033] Among them, the intrinsic orientation elements of the camera may include: lens distortion model parameters, lens focal length and camera principal point position; the extrinsic orientation elements of the camera may include: the coordinates of the camera principal point in the world coordinate system and the orientation of the camera main optical axis in the world coordinate system.
[0034] In another embodiment, the position of the auxiliary positioning point in the image is automatically identified, and the position of the auxiliary positioning point in the calibration field is obtained based on the position in the image; the position of the target point in the image is obtained based on the position of the auxiliary positioning point in the calibration field, the known world coordinates of the target point and the camera error model.
[0035] Specifically, the calibration device controls the camera to capture images including the auxiliary positioning points and the target points and acquires these images, and performs the following processing on each image.
[0036] In an image, there are usually multiple auxiliary positioning points, each of which has a unique number. First, automatically identify at least three auxiliary positioning points in the image and obtain the position of the auxiliary positioning points in the image. The position is represented by a two-dimensional coordinate (with the upper left corner of the image as the origin of the coordinate system), such as: b The two-dimensional position coordinates of the auxiliary positioning point are (x b ,y b ).
[0037] In one embodiment, the three-dimensional position coordinates of each target point in the calibration field in the world coordinate system are measured in advance, each target point is numbered, and the correspondence between the three-dimensional position coordinates of each target point and the number is stored in the calibration device, and the three-dimensional position coordinates of each auxiliary positioning point in the calibration field in the world coordinate system are measured in advance, each auxiliary positioning point is numbered, and the correspondence between the three-dimensional position coordinates of each auxiliary positioning point and the number is stored in the calibration device. The format of the correspondence between the numbers of the target points and the auxiliary positioning points and the corresponding three-dimensional position coordinates can be in the form of a collection as shown in the following example:
[0038]
[0039] Among them, M a is the target point, N b is the auxiliary positioning point, and the corresponding XYZ represent the three-dimensional position coordinates of the target point and the auxiliary positioning point respectively.
[0040] According to the correspondence between the numbers of the target points and the auxiliary positioning points and the corresponding three-dimensional position coordinates, the correspondence between the numbers of the target points and the auxiliary positioning points and the corresponding two-dimensional position coordinates can be obtained. The format can be in the following set form:
[0041]
[0042] Among them, m a is the target point, n b is the auxiliary positioning point, and the corresponding XY represents the two-dimensional position coordinates of the target point and the auxiliary positioning point respectively.
[0043] The two-dimensional position coordinates of the identified auxiliary positioning point are compared with the two-dimensional position coordinates of each auxiliary positioning point in the set p in the corresponding relationship between the two-dimensional position coordinates and the number, and the number of the auxiliary positioning point in the set p that is the same as or closest to the XY axis coordinates of the identified auxiliary positioning point is used as the number of the identified auxiliary positioning point. For example, the position coordinates of the identified auxiliary positioning point are (X b2 , Y b2 ), then the number of the identified auxiliary positioning point is b2, and the N b2 The three-dimensional position coordinates (X b2 , Y b2 , Z b2 ) as the three-dimensional position coordinates of the identified auxiliary positioning point.
[0044] In another embodiment, only the above-mentioned P set may be established. The two-dimensional position coordinates of the identified auxiliary positioning point are compared with the three-dimensional position coordinates of each auxiliary positioning point in the corresponding relationship between the three-dimensional position coordinates and the number, and the number of the auxiliary positioning point in the above-mentioned set P that is the same as or closest to the XY axis coordinates of the identified auxiliary positioning point is used as the number of the identified auxiliary positioning point. For example, the position coordinates of the identified auxiliary positioning point are (X b3 , Y b3 ), then according to the comparison with the three-dimensional position coordinates in the P set, the number of the identified auxiliary positioning point is obtained as b3, and the N in the P set is b3 The three-dimensional position coordinates (X b3 , Y b3 , Z b3 ) as the three-dimensional position coordinates of the identified auxiliary positioning point.
[0045] According to the identified 3D and 2D coordinates of the auxiliary positioning points, the direct linear transformation (DLT) method is used to construct a linear equation to calculate the initial pose of the camera corresponding to each image. According to the initial pose and the known world coordinates of the target point, combined with the estimated parameters of the error model of the camera lens, the 2D position coordinates of the target point on the image are fitted. The error model is used to characterize the deformation caused by the camera when shooting an image.
[0046] Specifically, the error model is a camera lens distortion model, specifically a Brown model.
[0047] The estimated parameters of the error model are also the estimated parameters of the lens distortion model, and the estimated parameters are roughly obtained according to the lens distortion model based on the identified auxiliary positioning points.
[0048] Further fitting a more accurate target point center. Since the camera cannot guarantee to shoot directly at all target points when taking photos, the target points will be more or less deformed. In the photos actually taken, the target points will appear as elliptical shapes. Therefore, it is necessary to project the deformed target points with a circle to obtain the accurate center of the target point. Specifically, extract a separate image of each target point from the image taken by the camera, use the edge detection method to detect the edge of the center circle of the target point in the separate image, and back-project the center circle of the target point with an elliptical edge into a circle to obtain the center coordinates of the circle. Then project the center coordinates onto the corresponding target point with an elliptical edge to obtain the pixel coordinates of the center of the target point in the image taken by the camera. According to the two-dimensional position coordinates of the target point on the image, the pixel coordinates of the center of the target point, and the lens distortion model of the camera, the lens distortion model parameters of the camera are calculated.
[0049] The edge detection method may specifically be a Canny edge detection method.
[0050] A single image of a target point means that there is only one target point in the image.
[0051] It should be noted that when DLT is used to construct linear equations, the object coordinate system (i.e., the world coordinate system) and the image coordinate system (i.e., the coordinate system on the image taken by the camera) form collinear equations. Since the three-dimensional coordinates of the target point on the object side are known, the projection relationship between the target point at the time of shooting and the target point in the image can be restored according to the known collinear equations, thereby interactively obtaining the position of the camera. According to the characteristics of the lens distortion error, the lens distortion model uses the Brown error model, and based on the error between the corresponding relationships of the projections, the distortion model parameters are inferred, and the accurate internal orientation elements of the camera including the focal length and the position of the principal point are calculated.
[0052] Compared with the estimated parameters of the above lens distortion model, the parameters of the lens distortion model calculated when calibrating the intrinsic orientation elements of the camera are more precise and can more accurately reflect the lens distortion.
[0053] S102, calibrating the interior orientation elements and exterior orientation elements of the laser radar according to the original three-dimensional point cloud acquired by the laser radar and the preset reference point cloud;
[0054] The exterior orientation elements of the laser radar and the exterior orientation elements of the camera are elements of the same coordinate system;
[0055] The internal orientation elements of the laser radar may include: correction coefficients of the laser radar ranging and angle measurement calibration model;
[0056] The exterior orientation elements of the laser radar may include: the coordinates of the center point of the laser radar in the world coordinate system and the conversion relationship between the laser radar coordinate system and the world coordinate system.
[0057] The original three-dimensional point cloud can be obtained by directly scanning the indoor environment of the calibration field with a laser radar, or it can be the original data of the indoor environment of the calibration field with a laser radar with a turntable and the turntable reading data;
[0058] The preset reference point cloud is a high-precision point cloud of the indoor environment of the calibration field acquired in advance and pre-configured in the calibration device. The coordinate system of the reference point cloud is the world coordinate system.
[0059] Specifically, in another embodiment, the calibration device extracts the point features of the original three-dimensional point cloud and the reference point cloud, and aligns the point features of the original three-dimensional point cloud and the reference point cloud through a point cloud registration algorithm to obtain a three-dimensional error between the point features, which is a three-dimensional position difference. By aligning the original three-dimensional point cloud and the reference point cloud, the position difference of the original three-dimensional point cloud relative to the reference point cloud is obtained, and based on a specially designed laser radar error model, according to the three-dimensional error between the point features, the error parameters of the laser radar internal orientation error model are calculated, that is, the error parameters of the original three-dimensional point cloud relative to the reference point cloud during laser radar ranging and angle measurement are calculated, that is, the correction coefficients of the laser radar ranging and angle measurement calibration model are obtained. It can be understood that the laser radar error model can also be an existing laser radar error model.
[0060] Furthermore, the internal orientation elements of the laser radar are calibrated according to the error parameters.
[0061] The point cloud registration algorithm may specifically be an iterative closest point (ICP, Iterative Closest Point) algorithm.
[0062] It should be noted that the origin of the original three-dimensional point cloud collected by the laser radar is the center of the laser radar, and the coordinate system is the laser radar coordinate system. By directly comparing the three-dimensional original point cloud with the reference point cloud in the world coordinate system, the position and orientation of the laser radar in the world coordinate system can be obtained, thereby obtaining the exterior orientation elements of the laser radar.
[0063] S103. Jointly calibrate the relative exterior orientation elements of the camera and the laser radar according to the relative relationship between the exterior orientation elements of the camera and the laser radar.
[0064] The relative relationship can be obtained based on the calibrated exterior orientation elements of the camera and the exterior orientation elements of the laser radar.
[0065] The relative external orientation elements of the camera and the laser radar include the relative positional relationship between the camera and the laser radar.
[0066] The exterior orientation elements of the camera and the laser radar are both in the world coordinate system. Based on the calibrated exterior orientation elements of the camera and the laser radar, the relative position relationship between the camera and the laser radar can be obtained, which can be the relative installation relationship, including the relative position deviation and angular installation deviation, etc. Based on the relative position relationship, the relative exterior orientation elements of the camera and the laser radar can be jointly calibrated.
[0067] In an embodiment of the present application, the auxiliary positioning points and target points in the calibration field are captured by a camera at one time, and the auxiliary positioning points and target points in the image can be automatically identified, thereby improving the efficiency and convenience of identifying data, so as to realize automatic calibration of the camera's internal orientation elements and external orientation elements, and compare the point cloud obtained by the laser radar with the reference point cloud, so as to realize fast and accurate calibration of the laser radar's internal orientation elements. Finally, according to the calibrated internal orientation elements and external orientation elements of the camera and laser radar, their internal orientation elements and relative external orientation elements are jointly calibrated. The relative external orientation elements characterize the relative position relationship between the camera and the laser radar, thereby realizing automatic, fast and accurate joint calibration of cameras and laser radars in the field of UAV surveying and mapping.
[0068] See also Figure 5 The embodiment of the present application also provides a device for joint calibration of a camera and a laser radar, which is Figure 1 The calibration device shown may be an electronic device including a computer having a data processing function corresponding to the data processing capability required in the embodiment of the present application, or may be a processor in the electronic device having the data processing function, such as a central processing unit (CPU) in a computer. The device includes: an identification module 201 and a calibration module 202.
[0069] The recognition module 201 is used to automatically recognize the auxiliary positioning points and the target points in the image of the calibration field taken by the camera;
[0070] A calibration module 202, configured to calibrate the internal and external orientation elements of the camera according to the recognition result;
[0071] The calibration module 202 is further used to calibrate the internal orientation elements and the external orientation elements of the laser radar according to the original three-dimensional point cloud obtained by the laser radar and the preset reference point cloud, and the external orientation elements of the camera and the external orientation elements of the laser radar are the same coordinate system elements;
[0072] The calibration module 202 is also used to jointly calibrate the relative external orientation elements of the camera and the laser radar according to the relative relationship between the external orientation elements of the camera and the external orientation elements of the laser radar. The relative external orientation elements of the camera and the laser radar include the relative position relationship between the camera and the laser radar.
[0073] Furthermore, the calibration module 202 is also used to extract point features of the original three-dimensional point cloud and the reference point cloud respectively;
[0074] Aligning the point features of the original three-dimensional point cloud with the point features of the reference point cloud to obtain a three-dimensional error between the point features, and calculating error parameters of the laser radar internal orientation error model based on the three-dimensional error between the point features using a pre-configured laser radar internal orientation error model;
[0075] The internal orientation elements of the laser radar are calibrated according to the error parameters.
[0076] The recognition module 201 is also used to automatically recognize the position of the auxiliary positioning point in the image, and obtain the position of the auxiliary positioning point in the calibration field according to the position in the image;
[0077] The position of the target point in the image is obtained according to the position of the auxiliary positioning point in the calibration field, the known world coordinates of the target point and the error model pre-configured in the camera.
[0078] The calibration module 202 is further used to extract a separate image of each target point from the image, and process each separate image to obtain the position of the center of each target point;
[0079] The lens distortion model parameters of the camera are calibrated according to the position of the target point in the image, the position of the center of the target point and the lens distortion model.
[0080] Furthermore, the recognition module 201 is also used to automatically recognize at least three auxiliary positioning points in the image, and obtain the two-dimensional position coordinates of the auxiliary positioning points in the image;
[0081] The number and three-dimensional position coordinates of the auxiliary positioning point are obtained based on the two-dimensional position coordinates of the auxiliary positioning point, the correspondence between the three-dimensional position coordinates of each auxiliary positioning point measured in advance and the number, and the correspondence between the two-dimensional position coordinates of each auxiliary positioning point set in advance and the number.
[0082] The recognition module 201 is further used to calculate the initial position and posture of the camera corresponding to the image according to the two-dimensional position coordinates and the three-dimensional position coordinates of the auxiliary positioning point by a direct linear transformation method;
[0083] According to the initial posture and the known world coordinates of the target point, combined with the estimated parameters of the lens distortion model of the camera, the two-dimensional position coordinates of the target point on the image are obtained.
[0084] The recognition module 201 is further used to extract a separate image of the target point from the image, and detect the edge of the central circle of the target point in the separate image;
[0085] Back-projecting the center circle of the target point whose edge is an ellipse into a circle, and determining the coordinates of the center of the circle;
[0086] The center coordinates of the circle are projected onto the corresponding target point whose edge is an ellipse to obtain the pixel coordinates of the center of the target point.
[0087] For details not included in the embodiments of the present application, please refer to the relevant description in the aforementioned embodiments.
[0088] In an embodiment of the present application, the device controls the camera to collect images of the auxiliary positioning points and target points in the calibration field at one time. The recognition module can automatically recognize the auxiliary positioning points and target points in the image, improve the efficiency and convenience of recognizing data, so as to realize automatic calibration of the internal orientation elements and external orientation elements of the camera. The calibration module can quickly and accurately calibrate the internal orientation elements of the laser radar by comparing the point cloud obtained by the laser radar with the reference point cloud. The calibration module jointly calibrates the internal orientation elements and relative external orientation elements of the calibrated camera and laser radar according to their internal orientation elements and external orientation elements. The relative external orientation elements characterize the relative position relationship between the camera and the laser radar, thereby realizing automatic, fast and accurate joint calibration of cameras and laser radars in the field of UAV surveying and mapping.
[0089] An embodiment of the present application also provides a system for joint calibration of a camera and a laser radar, including: a calibration device, a camera, a laser radar, and a calibration field provided with auxiliary positioning points and target points.
[0090] Among them, the calibration device is the above Figure 1 The calibration device 10 shown in the figure is the camera Figure 1 The camera 20 shown, the laser radar is the above Figure 1 The laser radar 30 shown in the figure has the auxiliary positioning point as mentioned above. Figure 1 The auxiliary positioning point 50 shown, the target point is the above Figure 1 The target point 60 shown in the figure is the calibration field mentioned above. Figure 1 The calibration field 40 is shown.
[0091] The calibration device is used to automatically identify the auxiliary positioning points and the target points in the image of the calibration field taken by the camera, and calibrate the internal orientation elements and external orientation elements of the camera according to the identification results; calibrate the internal orientation elements and external orientation elements of the laser radar according to the original three-dimensional point cloud acquired by the laser radar and the preset reference point cloud, and the external orientation elements of the camera and the external orientation elements of the laser radar are elements of the same coordinate system; jointly calibrate the relative external orientation elements of the camera and the laser radar according to the relative relationship between the external orientation elements of the camera and the external orientation elements of the laser radar, and the relative external orientation elements of the camera and the laser radar include the relative position relationship between the camera and the laser radar.
[0092] The image of each auxiliary positioning point is different, and each auxiliary positioning point has a center point.
[0093] For details not included in the embodiments of the present application, please refer to the relevant description in the aforementioned embodiments.
[0094] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0095] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0096] The above is a description of the method, device and system for joint calibration of the camera and lidar provided by the present invention. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for joint calibration of a camera and a laser radar, characterized in that: include: Automatically identify auxiliary positioning points and target points in an image of a calibration field taken by a camera, and calibrate the internal and external orientation elements of the camera according to the identification results; According to the original three-dimensional point cloud acquired by the laser radar and the preset reference point cloud, the internal orientation elements and the external orientation elements of the laser radar are calibrated, the external orientation elements of the camera and the external orientation elements of the laser radar are the same coordinate system elements, the original three-dimensional point cloud is obtained by the laser radar scanning the indoor environment of the calibration field, and the preset reference point cloud is a high-precision point cloud of the indoor environment of the calibration field acquired in advance; Jointly calibrating the relative exterior orientation elements of the camera and the laser radar according to the relative relationship between the exterior orientation elements of the camera and the laser radar, wherein the relative exterior orientation elements of the camera and the laser radar include the relative positional relationship between the camera and the laser radar; Wherein, the camera and the laser radar are respectively connected to a calibration device; The target points are set on the wall, the ground and the three-dimensional rack of the calibration field; The patterns and numbers of the auxiliary positioning points are different, and the positions and numbers of the auxiliary positioning points are obtained according to the center points of the patterns of the different auxiliary positioning points identified from the image of the calibration field; The automatic identification of auxiliary positioning points and target points in the image of the calibration field taken by the camera includes: Automatically identifying the position of the auxiliary positioning point in the image, and obtaining the position of the auxiliary positioning point in the calibration field according to the position in the image; Obtaining the position of the target point in the image according to the position of the auxiliary positioning point in the calibration field, the known world coordinates of the target point and the error model of the camera; The error model is a Brown model of a camera lens distortion model, and the step of calibrating the internal orientation elements of the camera according to the recognition result includes: Extracting a separate image of each target point from the image, and processing each separate image to obtain the position of the center of each target point; The lens distortion model parameters of the camera are calibrated according to the position of the target point in the image, the position of the center of the target point and the Brown model.
2. The method for joint calibration of a camera and a laser radar according to claim 1, characterized in that: The step of calibrating the internal orientation elements of the laser radar according to the original three-dimensional point cloud acquired by the laser radar and the preset reference point cloud includes: Extracting point features of the original three-dimensional point cloud and the reference point cloud respectively; Aligning the point features of the original three-dimensional point cloud with the point features of the reference point cloud to obtain a three-dimensional error between the point features, and calculating the error parameters of the laser radar internal orientation error model according to the three-dimensional error between the point features through the laser radar internal orientation error model; The internal orientation elements of the laser radar are calibrated according to the error parameters.
3. The method for joint calibration of a camera and a laser radar according to claim 1, characterized in that: The automatically identifying the position of the auxiliary positioning point in the image and obtaining the position of the auxiliary positioning point in the calibration field according to the position in the image includes: Automatically identify at least three auxiliary positioning points in the image, and obtain two-dimensional position coordinates of the auxiliary positioning points in the image; The number and three-dimensional position coordinates of the auxiliary positioning point are obtained based on the two-dimensional position coordinates of the auxiliary positioning point, the correspondence between the three-dimensional position coordinates of each auxiliary positioning point measured in advance and the number, and the correspondence between the two-dimensional position coordinates of each auxiliary positioning point set in advance and the number.
4. The method for joint calibration of a camera and a laser radar according to claim 3, characterized in that: The step of obtaining the position of the target point in the image according to the position of the auxiliary positioning point in the calibration field, the known world coordinates of the target point and the error model in the camera comprises: Calculating the initial position of the camera corresponding to the image according to the two-dimensional position coordinates and the three-dimensional position coordinates of the auxiliary positioning point by a direct linear transformation method; According to the initial posture and the known world coordinates of the target point, combined with the estimated parameters of the lens distortion model of the camera, the two-dimensional position coordinates of the target point on the image are obtained.
5. The method for joint calibration of a camera and a laser radar according to claim 4, characterized in that: The extracting a separate image of each target point from the image, and processing each separate image to obtain the position of the center of each target point includes: Extracting a separate image of the target point from the image, and detecting the edge of the central circle of the target point in the separate image; Back-projecting the center circle of the target point whose edge is an ellipse into a circle, and determining the coordinates of the center of the circle; The center coordinates of the circle are projected onto the corresponding target point whose edge is an ellipse to obtain the pixel coordinates of the center of the target point.
6. A device for joint calibration of a camera and a laser radar, characterized in that: The device is connected to a camera and a laser radar respectively, and the device comprises: A recognition module, used for automatically recognizing auxiliary positioning points and target points in the image of the calibration field taken by the camera; A calibration module, used for calibrating the internal orientation elements and the external orientation elements of the camera according to the recognition result; The calibration module is further used to calibrate the internal orientation elements and external orientation elements of the laser radar according to the original three-dimensional point cloud obtained by the laser radar and the preset reference point cloud, the external orientation elements of the camera and the external orientation elements of the laser radar are elements of the same coordinate system, the original three-dimensional point cloud is obtained by the laser radar scanning the indoor environment of the calibration field, and the preset reference point cloud is a high-precision point cloud of the indoor environment of the calibration field obtained in advance; The calibration module is further used to jointly calibrate the relative exterior orientation elements of the camera and the laser radar according to the relative relationship between the exterior orientation elements of the camera and the laser radar, wherein the relative exterior orientation elements of the camera and the laser radar include the relative position relationship between the camera and the laser radar; Wherein, the target points are set on the wall, the ground and the three-dimensional rack of the calibration field; The patterns and numbers of the auxiliary positioning points are different, and the positions and numbers of the auxiliary positioning points are obtained according to the center points of the patterns of the different auxiliary positioning points identified from the image of the calibration field; The automatic identification of auxiliary positioning points and target points in the image of the calibration field taken by the camera includes: Automatically identifying the position of the auxiliary positioning point in the image, and obtaining the position of the auxiliary positioning point in the calibration field according to the position in the image; Obtaining the position of the target point in the image according to the position of the auxiliary positioning point in the calibration field, the known world coordinates of the target point and the error model of the camera; The error model is a Brown model of a camera lens distortion model, and the step of calibrating the internal orientation elements of the camera according to the recognition result includes: Extracting a separate image of each target point from the image, and processing each separate image to obtain the position of the center of each target point; The lens distortion model parameters of the camera are calibrated according to the position of the target point in the image, the position of the center of the target point and the Brown model.
7. A system for joint calibration of a camera and a laser radar, characterized in that: include: Calibration equipment, cameras, laser radars and calibration fields with auxiliary positioning points and target points; The camera and the laser radar are respectively connected to a calibration device; The calibration device is used to automatically identify the auxiliary positioning point and the target point in the image of the calibration field taken by the camera, and calibrate the internal orientation elements and external orientation elements of the camera according to the recognition result; According to the original three-dimensional point cloud acquired by the laser radar and the preset reference point cloud, the internal orientation elements and the external orientation elements of the laser radar are calibrated, the external orientation elements of the camera and the external orientation elements of the laser radar are the same coordinate system elements, the original three-dimensional point cloud is obtained by the laser radar scanning the indoor environment of the calibration field, and the preset reference point cloud is a high-precision point cloud of the indoor environment of the calibration field acquired in advance; Jointly calibrating the relative exterior orientation elements of the camera and the laser radar according to the relative relationship between the exterior orientation elements of the camera and the laser radar, wherein the relative exterior orientation elements of the camera and the laser radar include the relative positional relationship between the camera and the laser radar; Wherein, the target points are set on the wall, the ground and the three-dimensional rack of the calibration field; The patterns and numbers of the auxiliary positioning points are different, and the positions and numbers of the auxiliary positioning points are obtained according to the center points of the patterns of the different auxiliary positioning points identified from the image of the calibration field; The automatic identification of the auxiliary positioning point and the target point in the image of the calibration field taken by the camera comprises: Automatically identifying the position of the auxiliary positioning point in the image, and obtaining the position of the auxiliary positioning point in the calibration field according to the position in the image; Obtaining the position of the target point in the image according to the position of the auxiliary positioning point in the calibration field, the known world coordinates of the target point and the error model of the camera; The error model is a Brown model of a camera lens distortion model, and the step of calibrating the internal orientation elements of the camera according to the recognition result includes: Extracting a separate image of each target point from the image, and processing each separate image to obtain the position of the center of each target point; The lens distortion model parameters of the camera are calibrated according to the position of the target point in the image, the position of the center of the target point and the Brown model.
8. The system for joint calibration of camera and laser radar according to claim 7, characterized in that: The image of each auxiliary positioning point is different, and each auxiliary positioning point has a center point.
