A Method for Joint Calibration of the Heading Angles of In-vehicle Millimeter-wave Radar and Camera

Abstract

The present invention discloses a method for jointly calibrating the heading angles of an in-vehicle millimeter-wave radar and a camera. By placing a radar reflector in front of the central normal line of an image calibration board perpendicular to the horizontal ground, driving the vehicle to be calibrated to a specified position to obtain the initial attitude parameters of the camera, and based on the detection result of the millimeter-wave radar on the radar reflector, combining the initial attitude parameters of the camera and the position of the radar reflector to correct the initial heading angle of the camera, obtaining the actual heading angle of the camera, completing the determination of the target azimuth of the camera, and effectively fusing it with the target output by the millimeter-wave radar. The present invention is carried out simultaneously with the sensor parameter calibration before the vehicle leaves the factory, accurately calibrating the heading angles of the millimeter-wave radar and the camera pointing to the same side of the vehicle, ensuring that the detection results of the millimeter-wave radar and the camera for the same target have the same heading angle, thereby improving the fusion success rate and accuracy of the detection information of these two sensors.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor joint calibration, and particularly to a method for jointly calibrating the heading angles of an in-vehicle millimeter-wave radar and a camera. Background Art

[0002] The control of vehicle autonomous driving usually requires fusing information from different types of sensors to obtain more accurate and reliable scene perception results. Among them, the visual perception information from the camera and the perception information from the millimeter-wave radar are the most commonly used fused information inputs. For a certain target in the scene, the camera and the millimeter-wave radar can provide independent perception results; the subsequent fusion module will identify the camera perception result and the millimeter-wave radar perception result corresponding to the target according to information such as the distance, azimuth, and speed of the target, and further perform information fusion. This requires the camera and the millimeter-wave radar to have consistent perception results (including distance, azimuth, and speed) for the same target. Therefore, before the vehicle leaves the factory, it is necessary to accurately calibrate the installation attitude of the camera and ensure that the camera and the radar have the same heading angle.

[0003] When calibrating the attitude of the camera, it is usually required that the vehicle body is directly facing the image calibration board, and at the same time, the vehicle center line is aligned with the center line of the image calibration board. However, in the actual operation process, if the vehicle alignment device is inaccurate or there is no precise vehicle alignment device, it is very difficult to park the vehicle directly facing, and there are usually errors in the heading angle of the camera obtained by camera calibration relative to the vehicle body central axis. When the deviation of the camera heading angle is large, there is a large deviation in the azimuth detection results of the millimeter-wave radar and the camera for the same target, resulting in the failure of feature fusion and reducing the overall performance of the system perception fusion. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for jointly calibrating the heading angles of an in-vehicle millimeter-wave radar and a camera for the deficiencies of the prior art, which can effectively solve the problem of target fusion failure caused by inconsistent calibrated heading angles of the millimeter-wave radar and the camera, reduce the requirements for the precise placement position of the vehicle, eliminate the need to introduce additional complex vehicle alignment devices, reduce the parameter calibration cost before the vehicle leaves the factory and the requirements for the measurement scene, and has a wider application range.

[0005] The technical solution for realizing the purpose of the present invention is as follows:

[0006] A method for jointly calibrating the heading angles of an in-vehicle millimeter-wave radar and a camera, comprising the following steps:

[0007] Step S1: Place the image calibration board, and the image calibration board is perpendicular to the horizontal ground;

[0008] Step S2: Place a radar reflector in front of the image calibration board on the central normal line of the image calibration board;

[0009] Step S3: Drive the vehicle to be calibrated to the calibration position in front of the radar reflector. The vehicle is equipped with a millimeter-wave radar and a camera. The calibration position requires that the camera can detect the image calibration board and can calibrate the attitude of the camera;

[0010] Step S4: Define with the plane where the image calibration board is located as the coordinate system to obtain the initial attitude parameters of the camera. When the vehicle to be calibrated is facing the image calibration board plane directly, the initial heading angle α of the camera is the actual heading angle, and proceed to the next step directly; when the vehicle to be calibrated is not facing the image calibration board plane directly, according to the detection result of the millimeter-wave radar on the radar reflector, combined with the initial attitude parameters of the camera and the position of the radar reflector, correct the initial heading angle of the camera to obtain the actual heading angle of the camera;

[0011] Step S5: Determine the target azimuth of the camera according to the actual heading angle of the camera and effectively fuse it with the target output by the millimeter-wave radar.

[0012] Further, in the step S4, when the vehicle to be calibrated is not facing the image calibration board plane directly, the actual heading angle of the camera is the heading angle β of the camera relative to the vehicle body central axis, and satisfies the following equation

[0013] β = α - δ

[0014] , where δ is the angle between the vehicle central axis and the image calibration board central axis, and α is the initial heading angle of the camera.

[0015] Further, the angle δ between the vehicle central axis and the image calibration board central axis satisfies the following equation

[0016] ,

[0017] where x is the lateral offset between the camera and the image calibration board, d2 is the straight-line distance between the millimeter-wave radar and the radar reflector, θ is the angle between the line connecting the radar reflector and the millimeter-wave radar and the central axis of the millimeter-wave radar, and L is the lateral deviation between the camera and the millimeter-wave radar

[0018] Further, the angle δ between the vehicle central axis and the image calibration board central axis is solved by the Levenberg-Marquardt algorithm for the following problem

[0019]

[0020] to obtain.

[0021] Further, in the step S4, an external parameter calibration method is adopted to obtain the initial attitude parameters of the camera.

[0022] Further, in the step S2, a laser aligner is used to determine the placement point of the radar reflector.

[0023] By adopting the above technical solution, the present invention has the following beneficial effects:

[0024] (1) When it is impossible to ensure that the vehicle is directly facing the image calibration board, the present invention corrects the initial heading angle of the camera to obtain the actual heading angle of the camera, thereby accurately calibrating the heading angles of the millimeter-wave radar and the camera pointing to the same side of the vehicle, effectively solving the problem of target fusion failure caused by inconsistent calibration heading angles between the millimeter-wave radar and the camera, reducing the requirement for the accurate placement position of the vehicle, eliminating the need to introduce additional complex vehicle alignment devices, reducing the parameter calibration cost before vehicle factory and the requirement for the measurement scenario, and having a wider application range.

[0025] (2) The present invention can accurately obtain the initial attitude parameters of the camera through the external parameter calibration method, ensuring the accuracy of the data and the precision of the heading angle correction.

[0026] (3) The present invention determines the placement point of the radar reflector through a laser aligner, thereby ensuring that the radar reflector is located on the central normal line of the image calibration board and ensuring the accuracy of the subsequent measurement data. Description of the Drawings

[0027] In order to make the content of the present invention easier to be clearly understood, the following further details the present invention according to specific embodiments in conjunction with the drawings, where:

[0028] Figure 1 is a flowchart of the heading angle joint calibration method of the present invention;

[0029] Figure 2 is a schematic diagram of the basic principle of vehicle factory calibration of the present invention;

[0030] Figure 3 is a schematic diagram of the heading angle correction principle of the present invention.

[0031] The reference numerals in the drawings are:

[0032] Vehicle 1, millimeter-wave radar 2, camera 3, radar reflector 4, image calibration board 5. Detailed Embodiment

[0033] In order to better understand the above technical solution, the following will detail the above technical solution in conjunction with the drawings of the specification and specific embodiments.

[0034] (Embodiment 1)

[0035] For various typical ADAS functions, the system usually needs to fuse information from millimeter-wave radars and cameras. The method of this embodiment is carried out simultaneously with the calibration of the sensor parameters before the vehicle leaves the factory, accurately calibrating the heading angles of the millimeter-wave radar and the camera pointing to the same side of the vehicle, ensuring that the detection results of the millimeter-wave radar and the camera for the same target have the same heading angle, thereby improving the fusion success rate and accuracy of the detection information of these two sensors.

[0036] Taking the in-vehicle forward-looking millimeter-wave radar and the in-vehicle forward-looking monocular camera as examples, the working process of the present invention is described. For millimeter-wave radars and cameras installed in other positions of the vehicle, the joint calibration method in the present invention can still be used.

[0037] As Figures 1 to 3 shown, this embodiment includes a vehicle 1, a millimeter-wave radar 2, a camera 3, a radar reflector 4, and an image calibration board 5. The millimeter-wave radar 2 and the camera 3 are installed at the front of the vehicle 1, and the lateral deviation L between the camera 3 and the millimeter-wave radar 2 is measured. Based on the above facilities, a method for jointly calibrating the heading angles of the in-vehicle millimeter-wave radar and the camera in this embodiment is constructed, which specifically includes the following steps:

[0038] Step S1: Place the image calibration board 3, requiring that the image calibration board 3 is perpendicular to the horizontal ground.

[0039] Step S2: Place the radar reflector 2 at a certain distance in front of the image calibration board 1, use a laser aligner to ensure that the radar reflector 4 is located on the normal line passing through the center of the image calibration board 5, and accurately measure the distance d0 between the radar reflector 4 and the center of the image calibration board through the laser aligner.

[0040] Step S3: Drive the vehicle 1 to be calibrated to a calibration position at a certain distance in front of the radar reflector 4, only need to ensure that it is basically facing the image calibration board 5. At the same time, the calibration position requires that the camera 3 can detect the image calibration board 5 and can calibrate the attitude of the camera 3.

[0041] Step S4: Define the coordinate system with the plane where the image calibration board 5 is located. Calculate the initial attitude parameters of the camera 3 through the Zhang Zhengyou calibration method, including the longitudinal straight-line distance d1 between the camera 3 and the image calibration board 5, the lateral offset x between the camera 3 and the image calibration board 5, and the initial heading angle α of the camera 3. When constructing the model of this embodiment, d0 is generally required to be slightly smaller than d1. Usually, d0 is 2 - 4 m smaller than d1. In this embodiment, 3 m is preferably used, so that the distance between the radar reflector 4 and the millimeter-wave radar 2 is about 3 m. The heading angle accuracy of the millimeter-wave radar 2 is relatively high within this range, ensuring the measurement accuracy of this method; the selection of d1 is related to the size of the image calibration board and the external parameter calibration algorithm, as long as it ensures that the camera 3 can perform attitude calibration, usually about 3 - 5 m.

[0042] When the vehicle 1 to be calibrated is exactly facing the plane of the image calibration board 5, the actual heading angle of the camera 3, that is, the heading angle β of the camera 3 relative to the vehicle body central axis, is equal to the initial heading angle α. The initial heading angle α of the camera 3 is the actual heading angle, and directly proceed to the next step.

[0043] When the vehicle 1 to be calibrated is not facing the plane of the image calibration board 5, according to the detection result of the millimeter-wave radar 2 on the radar reflector 4, combined with the initial attitude parameters of the camera 3 and the position of the radar reflector 4, correct the initial heading angle α of the camera 3 to obtain the actual heading angle of the camera 3, that is, the heading angle β of the camera 3 relative to the vehicle body central axis.

[0044] Specifically, since the radar reflector 4 will generate a very strong echo to the millimeter-wave signal irradiated on its surface, the millimeter-wave radar 2 can accurately and stably detect the position where the radar reflector 4 is located, and output the straight-line distance d2 between the millimeter-wave radar 2 and the radar reflector 4 and the angle θ between the connection line between the radar reflector 4 and the millimeter-wave radar 2 and the central axis of the millimeter-wave radar 2. According to the trigonometric geometric relationship, it can be known that

[0045] β = α - δ

[0046] where δ is the angle between the vehicle body central axis and the image calibration board central axis, and satisfies the following equation:

[0047]

[0048] δ is obtained by solving the following problem through the Levenberg - Marquardt algorithm:

[0049]

[0050] Finally, the angle δ between the vehicle body central axis and the image calibration board central axis is obtained, so as to obtain the accurate value of the heading angle β of the camera 3 relative to the vehicle body central axis, that is, the actual heading angle of the camera 3.

[0051] Step S5: Determine the target azimuth of the camera according to the actual heading angle of the camera 3, and effectively fuse it with the target output by the millimeter-wave radar 2.

[0052] Since it is necessary to ensure that the vehicle body is directly facing the image calibration board during calibration, and at the same time the vehicle center normal line is strictly aligned with the image calibration board center normal line, the traditional calibration method is to use a complex and expensive mechanical alignment device to automatically adjust the precise position of the vehicle. In some scenarios, it cannot be deployed due to limitations such as cost and space. In this embodiment, when it is impossible to ensure that the vehicle is directly facing the image calibration board 5, the initial heading angle of the camera is corrected to obtain the actual heading angle of the camera, so as to accurately calibrate the heading angles of the millimeter-wave radar and the camera pointing to the same side of the vehicle, effectively solve the problem of target fusion failure caused by the inconsistent calibration heading angles of the millimeter-wave radar and the camera, reduce the requirements for the precise placement position of the vehicle, eliminate the need to introduce additional complex vehicle alignment devices, reduce the parameter calibration cost before vehicle factory and the requirements for the measurement scenario, and have a wider application range.

[0053] The specific embodiments described above have further elaborated on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for jointly calibrating the heading angles of an in-vehicle millimeter-wave radar and a camera, characterized in that, Including the following steps: Step S1: Place the image calibration board, and the image calibration board is perpendicular to the horizontal ground; Step S2: Place the radar reflector in front of the image calibration board on the central normal line of the image calibration board; Step S3: Drive the vehicle to be calibrated to the calibration position in front of the radar reflector. The vehicle is equipped with a millimeter-wave radar and a camera. The calibration position requires that the camera can detect the image calibration board and can calibrate the attitude of the camera; Step S4: Define with the plane where the image calibration board is located as the coordinate system to obtain the initial attitude parameters of the camera. When the vehicle to be calibrated is facing the image calibration board plane directly, the initial heading angle α of the camera is the actual heading angle, and proceed to the next step directly; when the vehicle to be calibrated is not facing the image calibration board plane directly, according to the detection result of the millimeter-wave radar on the radar reflector, combined with the initial attitude parameters of the camera and the position of the radar reflector, correct the initial heading angle α of the camera to obtain the actual heading angle of the camera. Define the actual heading angle of the camera as the heading angle β of the camera relative to the central axis of the vehicle body, and satisfy the following equation β = α - δ, where α is the initial heading angle of the camera, and δ is the angle between the central axis of the vehicle and the central axis of the image calibration board, and satisfies the following equation ， where x is the lateral offset between the camera and the image calibration board, d2 is the straight-line distance between the millimeter-wave radar and the radar reflector, θ is the angle between the connection line between the radar reflector and the millimeter-wave radar and the central axis of the millimeter-wave radar, and L is the lateral deviation between the camera and the millimeter-wave radar; Step S5: Determine the target azimuth of the camera according to the actual heading angle of the camera and effectively fuse it with the target output by the millimeter-wave radar.

2. The vehicle-mounted millimeter-wave radar and camera heading angle joint calibration method according to claim 1, wherein: δ is obtained by solving the following problem through the Levenberg-Marquardt algorithm to get.

3. A method for jointly calibrating the heading angles of an in-vehicle millimeter-wave radar and a camera according to claim 1, characterized in that: In the said step S4, the initial attitude parameters of the camera are obtained by using the external parameter calibration method.

4. A method for jointly calibrating the heading angles of an in-vehicle millimeter-wave radar and a camera according to claim 1, characterized in that: In the said step S2, a laser aligner is used to determine the placement point of the radar reflector.

Images