Calibration method, device, electronic device and medium for laser radar and camera

Abstract

The present application provides a laser radar and camera calibration method, device, electronic device and medium, the method comprising: determining a first coordinate transformation relationship between an auxiliary camera and the laser radar; determining a second coordinate transformation relationship between a target camera and the auxiliary camera; determining a target coordinate transformation relationship through the first coordinate transformation relationship and the second coordinate transformation relationship; and determining external parameter data of the target camera and the laser radar based on the target coordinate transformation relationship. The laser radar and camera calibration method, device, electronic device and medium of the present application can ensure high accuracy of the calibration result when the laser radar and the camera are far apart.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a laser radar and camera calibration method, device, electronic device and computer-readable medium. Background Art

[0002] In the field of autonomous driving, using a variety of sensors to obtain accurate information about pedestrians and vehicles on the road, including speed, position, etc., has become an indispensable technical part. To achieve the above functions, image information, that is, two-dimensional information of the target object, is usually obtained through a camera; and depth distance information, that is, three-dimensional information of the target object, is obtained through a laser radar. If you want to obtain two-dimensional information and three-dimensional information of the target object at the same time, you need to calibrate the two sensors and fuse the data, or you can use a binocular sensor to obtain two-dimensional information and three-dimensional information at the same time. Among the two methods mentioned above, the former has a longer usage distance, so it is more suitable for the operation of autonomous driving equipment in complex environments. In this context, how to accurately calibrate cameras and laser radars at a long distance has become a technical problem that technicians must solve. To solve this technical problem, the existing technology uses a manual calibration method to obtain external parameter data of the laser radar and camera. Among them, the external parameter data describes how to convert points from the camera coordinate system to the laser radar coordinate system. For example, the prior art can use the MATLAB toolbox and manually extract the geometric plane of the calibration plate on the image and three-dimensional information to obtain its external parameter data in the camera coordinate system and the laser radar coordinate system, collect multiple sets of data by manual extraction, and then obtain the external parameters of the camera and the laser radar. Or collect the images and three-dimensional information of multiple calibration plates at one time and calibrate them uniformly.

[0003] When the distance between the laser radar and the camera is very close, the calibration results obtained by the existing technology have high accuracy. However, when the distance between the laser radar and the camera is far, the calibration results obtained by using the existing technology are difficult to meet the accuracy requirements. Summary of the invention

[0004] In view of this, the present application provides a laser radar and camera calibration method, device, electronic device and medium, which can ensure high accuracy of the calibration results when the laser radar and camera are far apart.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0006] According to a first aspect of an embodiment of the present application, a calibration method for a laser radar and a camera is provided, which includes: determining a first coordinate transformation relationship between an auxiliary camera and the laser radar; determining a second coordinate transformation relationship between a target camera and the auxiliary camera; determining a target coordinate transformation relationship through the first coordinate transformation relationship and the second coordinate transformation relationship; and determining external parameter data of the target camera and the laser radar based on the target coordinate transformation relationship.

[0007] In an exemplary embodiment of the present application, the external parameter data includes: a rotation matrix and a translation matrix.

[0008] In an exemplary embodiment of the present application, determining a first coordinate transformation relationship of an auxiliary camera and a laser radar includes: determining external parameter data of the auxiliary camera and the laser radar using the auxiliary camera coordinate system as a reference coordinate system; and acquiring the first coordinate transformation relationship based on the external parameter data.

[0009] In an exemplary embodiment of the present application, the auxiliary camera coordinate system is used as a reference coordinate system to determine the external parameter data of the auxiliary camera and the laser radar, including: using the auxiliary camera coordinate system as a reference coordinate system, and determining the external parameter data of the auxiliary camera and the laser radar through a target calibration method.

[0010] In an exemplary embodiment of the present application, determining a second coordinate transformation relationship between a target camera and an auxiliary camera includes: determining external parameter data of the target camera and the auxiliary camera using the target camera coordinate system as a reference coordinate system; and acquiring the second coordinate transformation relationship based on the external parameter data.

[0011] In an exemplary embodiment of the present application, the target camera coordinate system is used as a reference coordinate system to determine the external parameter data of the target camera and the auxiliary camera, including: using the target camera coordinate system as a reference coordinate system and using a stereo calibration method to determine the external parameter data of the target camera and the auxiliary camera.

[0012] In an exemplary embodiment of the present application, the distance between the auxiliary camera and the laser radar is smaller than the distance between the target camera and the laser radar.

[0013] According to a second aspect of an embodiment of the present application, a calibration device for a laser radar and a camera is proposed, wherein the device includes: a first determination module, configured to determine a first coordinate transformation relationship between an auxiliary camera and the laser radar; a second determination module, configured to determine a second coordinate transformation relationship between a target camera and an auxiliary camera; a third determination module, configured to determine a target coordinate transformation relationship through the first coordinate transformation relationship and the second coordinate transformation relationship; and a fourth determination module, configured to determine external parameter data of the target camera and the laser radar based on the target coordinate transformation relationship.

[0014] In an exemplary embodiment of the present application, the external parameter data includes: a rotation matrix and a translation matrix.

[0015] In an exemplary embodiment of the present application, the first determination module is configured to determine the external parameter data of the auxiliary camera and the lidar with the auxiliary camera coordinate system as the reference coordinate system; and obtain a first coordinate transformation relationship based on the external parameter data.

[0016] In an exemplary embodiment of the present application, the first determination module is configured to use the auxiliary camera coordinate system as a reference coordinate system and determine the external parameter data of the auxiliary camera and the lidar through a target calibration method.

[0017] In an exemplary embodiment of the present application, the second determination module is configured to determine the external parameter data of the target camera and the auxiliary camera with the target camera coordinate system as the reference coordinate system; and obtain the second coordinate transformation relationship based on the external parameter data.

[0018] In an exemplary embodiment of the present application, the second determination module is configured to use the target camera coordinate system as a reference coordinate system and use a stereo calibration method to determine the external parameter data of the target camera and the auxiliary camera.

[0019] In an exemplary embodiment of the present application, the distance between the auxiliary camera and the laser radar is smaller than the distance between the target camera and the laser radar.

[0020] According to the third aspect of an embodiment of the present application, an electronic device is proposed, comprising: one or more processors; a storage system for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the laser radar and camera calibration method described in any one of the above items.

[0021] According to a fourth aspect of an embodiment of the present application, a computer-readable medium is proposed, on which a computer program is stored, characterized in that when the program is executed by a processor, a calibration method for a laser radar and a camera as described in any one of the above items is implemented.

[0022] According to the laser radar and camera calibration method, system, electronic device and computer-readable medium of the present application, it is possible to ensure high accuracy of the calibration results when the laser radar and camera are far apart.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. The drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 The present invention is a flowchart of a method for calibrating a laser radar and a camera according to an exemplary embodiment.

[0026] Figure 2 The present invention is a flowchart of a method for calibrating a laser radar and a camera according to an exemplary embodiment.

[0027] Figure 3 The invention is a block diagram of a laser radar and camera calibration system according to an exemplary embodiment.

[0028] Figure 4 is a block diagram of a laser radar and camera calibration device according to an exemplary embodiment;

[0029] Figure 5 It is a block diagram of an electronic device for calibration of a laser radar and a camera according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0031] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention may be practiced while omitting one or more of the specific details, or other methods, components, systems, steps, etc. may be adopted. In other cases, known methods, systems, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0032] The accompanying drawings are only schematic illustrations of the present invention, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor systems and / or microcontroller systems.

[0033] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0034] Exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] Figure 1 FIG. 1 is a flow chart of a method for calibrating a laser radar and a camera according to an exemplary embodiment. Figure 1 The laser radar and camera calibration method shown in the figure can ensure high accuracy of the calibration result when the laser radar and camera are far apart. Figure 1 , the calibration method of the lidar and camera in the exemplary embodiment of the present application is described.

[0036] In step S110 , a first coordinate transformation relationship between the auxiliary camera and the laser radar is determined.

[0037] According to an embodiment of the present invention, the auxiliary camera coordinate system can be used as a reference coordinate system to determine the external parameter data of the auxiliary camera and the laser radar, and the first coordinate transformation relationship can be obtained based on the external parameter data.

[0038] In the embodiment of the present invention, the distance between the auxiliary camera and the laser radar is smaller than the distance between the target camera and the laser radar, ensuring that the auxiliary camera and the laser radar are close to each other, so that the external parameter data of the auxiliary camera and the laser radar and the first coordinate transformation formula have high accuracy. For example, the auxiliary camera can be placed above the laser radar to ensure that the auxiliary camera and the laser radar have a small relative distance.

[0039] According to the example embodiment, the auxiliary camera coordinate system, the target camera coordinate system and the laser radar coordinate system may also be obtained. Among them, the auxiliary camera (or target camera) coordinate system may, for example, include its camera optical center and X, Y, and Z axes, the X and Y axes may be parallel to the camera imaging plane, and the Z axis is also called the optical axis, which is perpendicular to the camera imaging plane; the distance from the camera optical center to the camera imaging plane is the focal length of the camera. After obtaining the above coordinate system, taking the first coordinate transformation relationship as an example, the relationship between the auxiliary camera coordinate system and the laser radar coordinate system can be obtained through this relationship.

[0040] According to an example embodiment, the auxiliary camera coordinate system can be used as a reference coordinate system to determine a first coordinate transformation relationship between the auxiliary camera and the laser radar. When the auxiliary camera coordinate system is used as a reference coordinate system, a transformation relationship from the auxiliary camera coordinate system to the laser radar coordinate system can be obtained. That is, in the first coordinate transformation relationship, the auxiliary camera coordinate system is the initial coordinate system, and the laser radar coordinate system is the target coordinate system.

[0041] According to an example embodiment, the first external parameter data of the auxiliary camera and the laser radar can be determined by a target calibration method. Among them, the target calibration method can be a manual calibration method. For example, the target calibration method can be implemented by the calibration toolbox of MATLAB. MATLAB is a commercial mathematical software produced by MathWorks, an American company, which is a high-level technical computing language and interactive environment for algorithm development, data visualization, data analysis and numerical calculation. MATLAB includes CameraCalibration Toolbox, i.e., a calibration toolbox. For example, the geometric plane of the calibration plate on the image and three-dimensional information can be manually extracted to obtain its external parameter data in the auxiliary camera coordinate system and the laser radar coordinate system, and the external parameter data of the auxiliary camera and the laser radar and the first coordinate transformation relationship can be obtained by collecting multiple sets of data.

[0042] In step S120 , a second coordinate transformation relationship between the target camera and the auxiliary camera is determined.

[0043] According to an embodiment of the present invention, the target camera and the auxiliary camera may be cameras of the same type to ensure that their internal parameter data are the same. The internal parameter data describes the projection relationship of the camera from the three-dimensional space to the two-dimensional image, which can be determined by the parameters of the camera.

[0044] According to an embodiment of the present invention, the target camera coordinate system can be used as a reference coordinate system to determine the external parameter data of the target camera and the auxiliary camera, and the second coordinate transformation relationship can be obtained based on the external parameter data. The second coordinate transformation relationship can describe the relationship between the auxiliary camera coordinate system and the target camera coordinate system.

[0045] According to an exemplary embodiment, when the target camera coordinate system is used as the reference coordinate system, a conversion relationship from the target camera coordinate system to the auxiliary camera coordinate system can be obtained. That is, in the second coordinate transformation relationship, the target camera coordinate system is the initial coordinate system, and the auxiliary camera coordinate system is the target coordinate system.

[0046] According to the example embodiment of the present invention, the stereo calibration method stereo can be used to determine the second external parameter data of the auxiliary camera and the target camera, and the second external parameter data of the target camera and the auxiliary camera and the second coordinate transformation relationship. Among them, the stereo calibration method can be used for the calibration of the binocular sensor. For example, stereo CameraCalibrator (stereo calibration method) can be entered in the command line of MATLAB, and the stereo calibration toolbox interface can be called to calibrate the auxiliary camera and the target camera through parameter settings to obtain the external parameter data of the target camera and the auxiliary camera, thereby obtaining the second coordinate transformation relationship.

[0047] In step S130, the target coordinate transformation relation is determined by the first coordinate transformation relation and the second coordinate transformation relation. The first coordinate transformation relation can describe the coordinate transformation method from the auxiliary camera coordinate system to the laser radar coordinate system, and the second coordinate transformation relation can describe the coordinate transformation method from the target camera coordinate system to the auxiliary camera coordinate system. By substituting them into the target coordinate transformation relation, the target coordinate transformation relation can be obtained, which can describe the coordinate transformation method from the target camera coordinate system to the laser radar coordinate system.

[0048] According to an example embodiment, the external parameter data may include: a rotation matrix and a translation matrix. The rotation matrix may describe the direction of the coordinate axis of the initial coordinate system relative to the coordinate axis of the target coordinate system in the coordinate transformation relationship, and the translation matrix may describe the position of the origin of the target coordinate system in the initial coordinate system in the coordinate transformation relationship.

[0049] Taking the first coordinate transformation relationship as an example, the relationship can be expressed as

[0050] L1=R1×O2+T1 (1)

[0051] Among them, L1 is the laser radar coordinate system, O2 is the auxiliary camera coordinate system, R1 is the rotation matrix in the first external parameter data, and T1 is the translation matrix in the first external parameter data.

[0052] Through the first coordinate transformation relationship, the mutual conversion relationship from the auxiliary camera coordinate system to the lidar coordinate system can be obtained.

[0053] Similarly, the second coordinate transformation relationship can be expressed as:

[0054] O2=R2×O1+T2 (2)

[0055] Among them, O2 is the auxiliary camera coordinate system, O1 is the target camera coordinate system, R2 is the rotation matrix in the second external parameter data, and T2 is the translation matrix in the second external parameter data. Through the second coordinate transformation relationship, the mutual conversion relationship from the target camera coordinate system to the auxiliary camera coordinate system can be obtained.

[0056] Finally, by substituting the second coordinate transformation relation (2) into the first coordinate transformation relation (1) and performing the combination, the target coordinate transformation relation can be obtained by replacing O2 in the first coordinate transformation relation with the second coordinate transformation relation. Based on the above example, the target coordinate transformation relation can be expressed as:

[0057] L1=R1×R2×O1+R1×T2+T1 (3)

[0058] Through the target coordinate transformation relationship, the mutual conversion relationship from the target camera coordinate system to the auxiliary camera coordinate system can be obtained.

[0059] In step S140, the external parameter data of the target camera and the laser radar are determined based on the target coordinate transformation relationship.

[0060] According to an embodiment of the present invention, the target coordinate transformation relational expression can describe the coordinate transformation relation between the target camera and the laser radar, so that the external parameter data of the target camera, the target camera and the laser radar can be determined accordingly. Determining the external parameter data of the target camera, the target camera and the laser radar is to calibrate the target camera and the laser radar.

[0061] According to an example embodiment, based on the external parameter data in the target coordinate transformation relationship, the external parameter data of the target camera and the laser radar can be determined, thereby realizing the calibration of the target camera and the laser radar. The external parameter data of the target camera and the laser radar may include a rotation matrix and a translation matrix from the target camera coordinate system to the laser radar coordinate system.

[0062] According to the laser radar and camera calibration method of the present application, an auxiliary camera is introduced and the auxiliary camera is used as an intermediate bridge to calibrate the target camera and laser radar at a long distance. Compared with the prior art that directly calibrates the target camera and laser radar at a long distance, in the present application, the distance between the auxiliary camera and the laser radar is set to be smaller than the distance between the target camera and the laser radar, and the external parameter data and the first coordinate transformation relationship based on the smaller distance between the auxiliary camera and the laser radar are more accurate than the larger distance between the target camera and the laser radar, and the external parameter data based on the calibration of the binocular camera are more accurate than the external parameter data and the second coordinate transformation relationship based on the larger distance between the target camera and the laser radar, so that the external parameter data of the target camera and the laser radar obtained by the first coordinate transformation relationship and the second relationship are more accurate than the external parameter data directly calibrated for the larger distance between the target camera and the laser radar, so that the high precision of the calibration result can be guaranteed when the laser radar and the camera are far apart.

[0063] Figure 2 FIG. 1 is a flow chart of a method for calibrating a laser radar and a camera according to an exemplary embodiment. Figure 2 , the calibration methods of LiDAR and camera may include:

[0064] In step S210, two cameras of the same type are selected, namely a target camera and an auxiliary camera. The auxiliary camera is placed on the laser radar to ensure that it has a small relative distance with the laser radar, and the target camera can be placed at its target installation position. In this exemplary embodiment, the coordinate systems of the auxiliary camera, the target camera and the laser radar can be obtained first. The composition of the coordinate system has been explained in the aforementioned exemplary embodiment and will not be repeated here. For ease of expression, the coordinate systems of the auxiliary camera, the target camera and the laser radar can be expressed as O1, O2 and L1, respectively.

[0065] In step S220, the auxiliary camera and the laser radar are calibrated. Among them, the calibration toolbox of MATLAB can be selected for manual selection and calibration. Taking the auxiliary camera coordinate system as the reference coordinate system, the external parameter data (R1, T1) of the auxiliary camera and the laser radar are calibrated manually through MATLAB, so as to obtain the first coordinate transformation relationship. The first coordinate transformation relationship can be expressed as:

[0066] L1=R1×O2+T1 (1)

[0067] In step S230, the target camera and the auxiliary camera are calibrated. Among them, the MATLAB stereo calibration method can be selected, and the target camera coordinate system is used as the reference coordinate system. The MATLAB stereo calibration obtains the external parameter data (R2, T2) of the binocular camera, thereby obtaining the second coordinate transformation relationship. The second coordinate transformation relationship can be expressed as:

[0068] O2=R2×O1+T2 (2)

[0069] In step S240, the external parameters of the target camera and the laser radar are calculated by coordinate transformation. The first coordinate transformation relationship and the second coordinate transformation relationship can be combined to obtain the target coordinate transformation relationship. The target coordinate transformation relationship can describe the coordinate transformation relationship between the target camera and the laser radar. The target coordinate transformation relationship can be expressed as:

[0070] L1=R1×R2×O1+R1×T2+T1 (3)

[0071] Among them, by decomposing the target coordinate transformation relationship, the target external parameter data (R3, T3) can be obtained, and the specific decomposition method is as follows:

[0072] R3=R1×R2 (4)

[0073] T3=R1×T2+T1 (5)

[0074] Among them, R3 is the rotation matrix in the target extrinsic parameter data, T3 is the translation matrix in the target extrinsic parameter data, and the extrinsic parameter data of the target camera and the laser radar are determined, thereby realizing the calibration of the target camera and the laser radar.

[0075] According to the laser radar and camera calibration method of the present application, an auxiliary camera is introduced and the auxiliary camera is used as an intermediate bridge to calibrate the target camera and laser radar at a long distance. Compared with the prior art that directly calibrates the target camera and laser radar at a long distance, in the present application, the distance between the auxiliary camera and the laser radar is set to be smaller than the distance between the target camera and the laser radar, and the external parameter data and the first coordinate transformation relationship based on the smaller distance between the auxiliary camera and the laser radar are more accurate than the larger distance between the target camera and the laser radar, and the external parameter data based on the calibration of the binocular camera are more accurate than the external parameter data and the second coordinate transformation relationship based on the larger distance between the target camera and the laser radar, so that the external parameter data of the target camera and the laser radar obtained by the first coordinate transformation relationship and the second relationship are more accurate than the external parameter data directly calibrated for the larger distance between the target camera and the laser radar, thereby improving the accuracy of the calibration result when the target camera and the laser radar are far apart.

[0076] Figure 3 is a block diagram of a laser radar and camera calibration system according to an exemplary embodiment. Figure 3 The laser radar and camera calibration system may include: a target camera 310, a laser radar 320, and an auxiliary camera 330.

[0077] In the laser radar and camera calibration system, the target camera 310 has a target camera coordinate system, for example, represented by O1. The laser radar 320 has a laser radar coordinate system, for example, represented by O2. The distance between the target camera 310 and the laser radar 320 can be determined according to actual conditions, as long as the distance between the auxiliary camera 330 and the laser radar 320 is less than the distance between the target camera 310 and the laser radar 320. The technical solution of the present invention does not specifically limit this.

[0078] The auxiliary camera 330 has an auxiliary camera coordinate system, for example, represented by L1. The auxiliary camera 330 can be used to determine the external parameter data of the auxiliary camera and the laser radar and the first coordinate transformation relationship; and determine the external parameter data of the auxiliary camera and the target camera and the second coordinate transformation relationship. Among them, it should be ensured that the auxiliary camera 330 and the laser radar 320 are close to each other to ensure that the calibration results, i.e., the external parameter data of the auxiliary camera and the laser radar and the first coordinate transformation have high accuracy. For example, the auxiliary camera 330 can be placed above the laser radar 320 to ensure that the auxiliary camera 330 and the laser radar 320 have a small relative distance.

[0079] According to an example embodiment, the auxiliary camera 330 can be used to determine the external parameter data of the auxiliary camera 330 and the laser radar 320 and the first coordinate transformation relationship using the auxiliary camera coordinate system as the reference coordinate system. The first external parameter data of the auxiliary camera 330 and the laser radar 320 and the first coordinate transformation relationship can be determined by a target calibration method. For example, the external parameter data of the auxiliary camera coordinate system and the laser radar coordinate system can be obtained by manually extracting the geometric plane of the calibration plate on the image and three-dimensional information, and the external parameter data of the auxiliary camera 330 and the laser radar 320 and the first coordinate transformation relationship can be obtained by collecting multiple sets of data.

[0080] According to an example embodiment, the auxiliary camera 330 can be used to determine the external parameter data of the target camera 310 and the auxiliary camera 330 and the second coordinate transformation relationship using the target camera coordinate system as the reference coordinate system. Among them, the external parameter data of the target camera 310 and the auxiliary camera 330 and the second coordinate transformation relationship can be determined using a stereo calibration method. Among them, the stereo calibration method can be used for the calibration of a binocular sensor. For example, the stereo Camera Calibrator command can be entered in the command line of MATLAB to call the stereo calibration toolbox interface, and the target camera 310 and the auxiliary camera 330 are calibrated through parameter settings to obtain the second external parameter data and the second coordinate transformation relationship.

[0081] According to an example embodiment, the laser radar and camera calibration system may further include a calibration module for determining a target coordinate transformation relationship through the first coordinate transformation relationship and the second coordinate transformation relationship; and determining the external parameter data of the target camera and the laser radar through the target coordinate transformation relationship to calibrate the target camera and the laser radar. The target external parameter data may include a rotation matrix and a translation matrix from the target camera coordinate system to the laser radar coordinate system. The rotation matrix and the translation matrix can be used to describe the relationship between the target camera coordinate system and the laser radar coordinate system, so that the target camera and the laser radar can be calibrated.

[0082] According to the laser radar and camera calibration system of the present application, an auxiliary camera is introduced and used as an intermediate bridge to calibrate the target camera and laser radar at a long distance. The laser radar and camera calibration system of the present application can ensure high accuracy of the calibration result when the laser radar and camera are far apart.

[0083] Figure 4 FIG. 1 is a block diagram of a laser radar and camera calibration device according to an exemplary embodiment. Figure 4 , the device 500 comprises:

[0084] The first determination module 410 is configured to determine a first coordinate transformation relationship between the auxiliary camera and the laser radar.

[0085] The second determination module 420 is configured to determine a second coordinate transformation relationship between the target camera and the auxiliary camera.

[0086] The third determination module 430 is configured to determine a target coordinate transformation relationship through the first coordinate transformation relationship and the second coordinate transformation relationship.

[0087] The fourth determination module 440 is configured to determine the external parameter data of the target camera and the laser radar based on the target coordinate transformation relationship.

[0088] According to the laser radar and camera calibration method of the present application, an auxiliary camera is introduced and the auxiliary camera is used as an intermediate bridge to calibrate the target camera and laser radar at a long distance. Compared with the prior art that directly calibrates the target camera and laser radar at a long distance, in the present application, the distance between the auxiliary camera and the laser radar is set to be smaller than the distance between the target camera and the laser radar, and the external parameter data and the first coordinate transformation relationship based on the smaller distance between the auxiliary camera and the laser radar are more accurate than the larger distance between the target camera and the laser radar, and the external parameter data based on the calibration of the binocular camera are more accurate than the external parameter data and the second coordinate transformation relationship based on the larger distance between the target camera and the laser radar, so that the external parameter data of the target camera and the laser radar obtained by the first coordinate transformation relationship and the second relationship are more accurate than the external parameter data directly calibrated for the larger distance between the target camera and the laser radar, thereby improving the accuracy of the calibration result when the target camera and the laser radar are far apart.

[0089] Figure 5 It is a block diagram of an electronic device for calibration of a laser radar and a camera according to an exemplary embodiment.

[0090] Refer to the following Figure 5 The electronic device 500 according to this embodiment of the present application is described. Figure 5 The electronic device 500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 to a random access memory (RAM) 503. For example, the central processing unit 501 can perform the following operations: Figure 1 , Figure 2 One or more of the steps shown.

[0092] In the RAM 503, various programs and data required for system operation are also stored, such as the auxiliary camera coordinate system, the target camera coordinate system, and external reference data, etc. The CPU 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0093] The following components are connected to the I / O interface 505: an input section 506 including a touch screen, a keyboard, etc.; an output section 507 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 508 including a flash memory, etc.; and a communication section 509 including a wireless network card, a high-speed network card, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 411, such as a semiconductor memory, a magnetic disk, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0094] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution of the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for a computing device (which can be a personal computer, a server, a mobile terminal, or a smart device, etc.) to execute the method according to the embodiment of the present invention, for example Figure 1 , Figure 2 One or more of the steps shown.

[0095] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0096] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not claimed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

[0097] It should be understood that the present invention is not limited to the detailed structures, drawings or implementations shown herein, but rather the present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A laser radar and camera calibration method, characterized in that: include: Taking the auxiliary camera coordinate system as the reference coordinate system, determining the first external parameter data of the auxiliary camera and the laser radar; Determine a first coordinate transformation relationship between the auxiliary camera and the laser radar based on the first external parameter data, where the first external parameter data includes a rotation matrix and a translation matrix; Taking the target camera coordinate system as the reference coordinate system, determining second extrinsic parameter data of the target camera and the auxiliary camera, and determining a second coordinate transformation relationship between the target camera and the auxiliary camera based on the second extrinsic parameter data; wherein the distance between the auxiliary camera and the laser radar is less than the distance between the target camera and the laser radar; Determine a target coordinate transformation relationship through the first coordinate transformation relationship and the second coordinate transformation relationship; and The third external parameter data of the target camera and the laser radar is determined based on the target coordinate transformation relationship, the rotation matrix of the third external parameter data is obtained by the rotation matrix of the first external parameter data and the rotation matrix of the second external parameter data, and the translation matrix of the third external parameter data is obtained by the rotation matrix of the first external parameter data, the translation matrix of the first external parameter data and the translation matrix of the second external parameter data.

2. The method according to claim 1, characterized in that Taking the auxiliary camera coordinate system as the reference coordinate system, the first external parameter data of the auxiliary camera and the laser radar are determined, including: The auxiliary camera coordinate system is used as the reference coordinate system, and the first external parameter data of the auxiliary camera and the lidar are determined by the target calibration method.

3. The method according to claim 1, characterized in that Taking the target camera coordinate system as the reference coordinate system, the second external parameter data of the target camera and the auxiliary camera are determined, including: Taking the target camera coordinate system as the reference coordinate system, the second extrinsic parameter data of the target camera and the auxiliary camera are determined using the stereo calibration method.

4. A laser radar and camera calibration device, characterized in that: The device comprises: A first determination module is configured to determine first external parameter data of the auxiliary camera and the laser radar using the auxiliary camera coordinate system as a reference coordinate system; determine a first coordinate transformation relationship between the auxiliary camera and the laser radar based on the first external parameter data, wherein the first external parameter data includes a translation matrix and a rotation matrix; a second determination module, configured to determine second extrinsic parameter data of the target camera and the auxiliary camera with the target camera coordinate system as a reference coordinate system, and determine a second coordinate transformation relationship between the target camera and the auxiliary camera based on the second extrinsic parameter data; wherein the distance between the auxiliary camera and the laser radar is less than the distance between the target camera and the laser radar; A third determining module is configured to determine a target coordinate transformation relationship through the first coordinate transformation relationship and the second coordinate transformation relationship; and The fourth determination module is configured to determine the third external parameter data of the target camera and the laser radar based on the target coordinate transformation relationship, the rotation matrix of the third external parameter data is obtained by the rotation matrix of the first external parameter data and the rotation matrix of the second external parameter data, and the translation matrix of the third external parameter data is obtained by the rotation matrix of the first external parameter data, the translation matrix of the first external parameter data and the translation matrix of the second external parameter data.

5. An electronic device, characterized in that: include: one or more processors; a storage system for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 3.

6. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images