Method for calibrating a camera and / or lidar sensor for a vehicle or robot

Abstract

The invention relates to a method for calibrating a camera (3) and / or a lidar sensor (2.1) of a vehicle (1) or a robot, wherein an image (B) of the vehicle environment or the robot environment is acquired by means of a camera (3), a real pattern (M) is converted by means of at least one lidar sensor (2.1) r ) is emitted to a vehicle environment or a robot environment within at least a portion of a camera (3) acquisition area, and the real pattern (M) is acquired by means of the camera (3) r According to the invention, a virtual pattern (M) is generated in the coordinate system of the laser radar sensor (2.1) by means of a laser radar sensor (2.1). v ) is projected onto a virtual plane (E v ), the laser beam emitted by the laser radar sensor (2.1) passes through the virtual plane (E v ) and generates a virtual pattern (M) on a real projection surface (A) v ) is the associated real pattern (M r ), the real pattern (M) captured by the camera (3) r ) is calculated based on the surface orientation of the real projection surface (A) to the virtual plane (E v ), thereby generating a corrected virtual pattern (M) in the coordinate system of the camera (3) ev ), and by placing the virtual pattern (M v ) and the corrected virtual pattern (M ev ) to calibrate the camera (3) and / or the lidar sensor (2.1).

Description

Technical Field

[0001] The invention relates to a method for calibrating a camera and / or a lidar sensor of a vehicle or a robot. Background Art

[0002] DE 10 2016 009 327 A1 discloses a method for calibrating a vehicle camera, wherein an image of the vehicle environment is captured by means of a camera. By means of at least one lidar sensor, a defined pattern is emitted into the vehicle environment within at least a portion of the camera's capture range, and the camera captures the pattern. By means of an evaluation unit connected to the camera or integrated therein, a distance value to the pattern contained in the image captured by means of the camera can be determined, and the camera is calibrated as a function of the distance value. The lidar sensor is designed to be self-calibrating, wherein the self-calibration is based on object tracking. Summary of the invention

[0003] The invention is based on the object of specifying a method for calibrating a camera and / or lidar sensor of a vehicle or a robot which is improved compared to the prior art.

[0004] According to the invention, this object is achieved by a method having the features described below.

[0005] Advantageous embodiments of the invention are the subject of the description below.

[0006] In a method for calibrating a camera and / or a lidar sensor of a vehicle or a robot, an image of the vehicle environment or the robot environment is captured by means of a camera. By means of at least one lidar sensor, a real pattern is emitted into the vehicle environment or the robot environment within at least a part of the camera capture range, wherein the real pattern is captured by means of the camera.

[0007] According to the present invention, a virtual pattern generated in the laser radar sensor coordinate system is projected into a virtual plane in the vehicle environment or robot environment by means of a laser radar sensor. Here, the laser beam emitted by the laser radar sensor passes through the virtual plane and generates a real pattern associated with / corresponding to the virtual pattern on a real projection surface. The real pattern captured by the camera is back-calculated into the virtual plane based on the surface direction of the real projection surface. As a result, a corrected virtual pattern is generated in the coordinate system of the camera, and the camera and / or the laser radar sensor are calibrated based on the comparison of the virtual pattern and the corrected virtual pattern.

[0008] In the images captured by the camera, optical distortions may occur due to the lens system and due to potential overlap of the camera. In order to avoid such optical distortions, the camera is usually calibrated during vehicle or robot production, for example using a checkerboard pattern. The correction values ​​determined are stored. However, the correctness of the correction values ​​may be adversely affected by aging of the components, different temperature ranges and external mechanical influences.

[0009] The method allows the camera to be calibrated for almost any object, so that calibration during the production process of the vehicle or robot, for example at the end of the production process, can be simplified. In addition, the calibration can also be carried out during the operation of the vehicle or robot, for example during short work breaks. Therefore, the possibility of calibration during the entire operation of the camera can also be realized, so as to also detect aging effects. Therefore, calibration can also be carried out and compensated during the operation of the camera and lidar sensor, which is also advantageous in particular in automated and in particular highly automated, autonomously or semi-autonomously operated vehicles. Multiple calibrations can also be combined in one step. The result of the calibration is that the camera and lidar sensor recognize the same object at the same location.

[0010] The robot is also designed as a vehicle, for example, a highly automated or fully automated car, a highly automated or fully automated transportation vehicle, or a highly automated or fully automated truck. The robot can also be an industrial robot, an automated lawn mower, a vacuum cleaning robot, a sweeping and mopping robot, or an automated surface transportation vehicle.

[0011] In a possible design of the method, the surface orientation of the real projection surface is determined according to the distance from the image point of the real pattern determined by means of a laser radar sensor. This allows the camera and / or the laser radar sensor to be calibrated in a so-called cross calibration, in which the reflection of the laser radar sensor is recorded in the camera and used to calibrate the camera relative to the laser radar sensor (i.e., with respect to the error between the camera and the laser radar sensor). Based on the detected laser radar reflection in the camera, a virtual pattern can be drawn up for any object in the environment of the vehicle or robot, which is measured stereoscopically by the laser radar sensor, so that the so-called ground truth can be generated. The relative calibration of the camera and the laser radar sensor relative to each other allows the objects identified by means of the camera and the laser radar sensor to be better matched to each other and to improve the accuracy. When the camera and the laser radar sensor are directly calibrated with each other, the relative error between them is reduced. By calibrating the camera directly with respect to the laser radar sensor and directly calibrating in the opposite way, the raw data of the camera and the laser radar sensor can be directly fused. In addition, the fusion of the independent data of the camera and the laser radar sensor is also improved.

[0012] In another possible design of the method, when comparing the virtual pattern and the corrected virtual pattern, at least one transformation formula for converting the virtual pattern into the corrected virtual pattern and / or converting the corrected virtual pattern into the virtual pattern is determined. That is, transformation parameters for transforming one virtual pattern into the other virtual pattern are calculated based on the two virtual patterns. The transformation parameters can be used to transform the environmental data determined by one of the sensor devices, i.e. the camera or the lidar sensor, into the coordinate system of the other sensor device.

[0013] In another possible design of the method, the azimuth error and / or elevation error of the camera and / or lidar sensor is determined in the calibration. The camera can thus be calibrated relative to the lidar sensor in such a way that possible differences between the optical axis of the camera and the optical axis of the lidar sensor, i.e. the azimuth error and elevation error, can be compensated. Displacements and twists of the lidar sensor coordinate system relative to the camera coordinate system, which are caused by the azimuth error and elevation error, can thus be determined. The displacements and twists are taken into account when comparing two virtual patterns that are also relatively displaced and twisted. Thus, a very accurate calibration can be performed.

[0014] In another possible embodiment of the method, in order to generate the virtual pattern, the laser beam emitted by the laser radar sensor is deflected by means of a swiveling mirror of the laser radar sensor. Thus, the virtual pattern and the real pattern in the virtual plane can be generated very simply and reliably.

[0015] In another possible embodiment of the method, when capturing the real pattern with the aid of the camera, an integration of the multiple images captured in succession by the camera is performed. By means of the integration, the camera resolution in the infrared range is increased, which is advantageous because cameras in the form of conventional color cameras have their highest sensitivity in the visible light range and a relatively low sensitivity in the infrared range. In addition, synchronous shaking of the swiveling mirrors can lead to variations in the distance between the lines of the virtual pattern and thus the lines of the real pattern. The integration of the multiple images captured in succession by the camera allows compensation of these errors.

[0016] In another possible embodiment of the method, the camera is switched into a calibration mode for calibration.

[0017] In another possible embodiment of the method, the virtual pattern is generated by means of infrared laser beams, so that it can be captured by means of a camera.

[0018] In another possible embodiment of the method, in order to generate the image captured by the camera, the laser light that hits the camera is filtered by means of the camera's own infrared filter / infrared filter. The infrared filter is used to reduce interference and improve color quality.

[0019] In another possible design of the method, an infrared filter is used, which can transmit the infrared laser rays emitted by the lidar sensor and reflected by the real projection surface, so that the real pattern can be captured.

[0020] In another possible design of the method, the infrared filter is switched in the calibration mode of the camera to be transparent to the infrared laser beam emitted by the lidar sensor and reflected by the real projection surface. Thus, the infrared filter is used to reduce interference and improve color quality during normal operation of the camera and is deactivated in the calibration mode to achieve optimized detection of the infrared laser beam emitted by the lidar sensor and reflected by the projection surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The embodiments of the present invention are explained in detail below with reference to the drawings, wherein:

[0022] Figure 1 A perspective view schematically showing a vehicle, a virtual pattern and a real pattern,

[0023] Figure 2 A block diagram of a device for calibrating a camera and / or lidar sensor of a vehicle is schematically shown. DETAILED DESCRIPTION

[0024] Parts that correspond to one another are provided with the same reference symbols in all the figures.

[0025] exist Figure 1 The vehicle 1 and the virtual pattern M are shown in v and real pattern M r perspective view. Figure 2 A block diagram of a possible embodiment of a device 4 for calibrating a camera 3 and / or a lidar sensor 2 . 1 of a vehicle 1 is shown.

[0026] The following description can also be applied similarly to a robot comprising at least one camera 3 and / or at least one lidar sensor 2.1. Such a robot can also be designed as a vehicle, such as a highly automated or fully automated car, a highly automated or fully automated transport vehicle or a highly automated or fully automated truck. The robot can also be an industrial robot, an automated lawn mower, a vacuum cleaning robot, a sweeping and mopping robot or an automated water surface transport vehicle.

[0027] The vehicle 1 comprises a lidar 2 having at least one lidar sensor 2 . 1 and a camera 3 , wherein the lidar 2 and the camera 3 are designed to capture the vehicle surroundings.

[0028] The device 4 includes a laser radar 2 , a camera 3 and a processing unit 5 .

[0029] The goal of the calibration is to calibrate camera 3 relative to lidar sensor 2.1 in such a way that possible differences between the optical axis of camera 3 and the optical axis of lidar sensor 2.1, i.e. azimuth errors and elevation errors, are compensated so that camera 3 and lidar sensor 2.1 recognize the same objects at the same location.

[0030] To this end, a virtual plane E is defined in front of the vehicle 1 v , which is located in the acquisition area of ​​the camera 3 and the laser radar sensor 2.1. With the help of the laser radar sensor 2.1, infrared laser rays are emitted, thereby v A virtual pattern M is generated on v . Virtual pattern M v For example, a chessboard pattern. v Here, it is generated in the coordinate system of the lidar sensor 2.1. The infrared laser beam passes through the virtual plane E v , so that the virtual pattern M v Related real pattern M r is projected onto a projection surface A in the vehicle environment, for example onto the road surface. r Relative to the virtual pattern M v Distortion, because the real pattern appears r The projection plane A is not parallel to the virtual plane E v .

[0031] In one possible embodiment, the lidar 2 or the lidar sensor 2 . 1 has a pivoting mirror, by means of which the infrared laser beam is deflected onto the scene to be scanned.

[0032] With the help of the LiDAR sensor 2.1, the distance from the projected real pattern M r The distances d of the individual image points of the projection surface A are thus determined in stereo by means of the processing unit 5, producing a so-called ground truth.

[0033] With the help of camera 3, the real pattern M is collected r At least one image B of the projection surface A. Based on the determined surface orientation of the projection surface A, the captured real pattern M v The processing unit 5 is used to correct the following: v The inverse calculation result is the corrected virtual pattern M calculated in the coordinate system of the camera 3. ev .

[0034] The synchronous shaking of the rotating mirror of the laser radar 2 may cause the virtual pattern M v The distance between the lines and thus the real pattern M rThe distance between the lines changes, in order to collect the real pattern M r In order to compensate for this error, in one possible design, a plurality of images B of the camera 3 are integrated. By means of the integration, the infrared resolution of the camera 3 is also increased, which is advantageous because the camera 3 designed as a conventional color camera has its highest sensitivity in the visible light range and a relatively low sensitivity in the infrared range.

[0035] In another possible design, the camera 3 is switched to a calibration mode for calibration. It is also possible that the camera 3 has an infrared filter to reduce interference and / or improve color quality. The infrared filter is designed, for example, so that it is transparent to reflected infrared laser pulses or it can be switched to a state in the calibration mode of the camera 3 so that it is transparent to the infrared laser beam of the lidar sensor 2.1.

[0036] Because of the azimuth and elevation errors, the coordinate system of the laser radar sensor 2.1 is displaced and distorted relative to the coordinate system of the camera 3. Therefore, the two virtual patterns M v 、M ev Also relative displacement and distortion.

[0037] From two virtual patterns M v 、M ev The method for converting a virtual pattern M v 、M ev Convert to another virtual pattern M ev 、M v That is, the transformation parameter P for coordinate transformation is determined, by means of which the data collected by the camera 3 can be transformed into the coordinate system of the laser radar sensor 2.1 or the data collected by the laser radar sensor 2.1 can be transformed into the coordinate system of the camera 3.

[0038] The surroundings data determined by lidar sensor 2 . 1 or camera 3 are then transformed during normal operation into the coordinate system of the respective other sensor using the determined transformation parameters P. Lidar sensor 2 . 1 and camera 3 then recognize the same objects at the same location.

Claims

1. A method for calibrating a camera (3) and / or a lidar sensor (2.1) of a vehicle (1) or a robot, wherein: - using the camera (3) to capture an image (B) of the vehicle environment or the robot environment, - using at least one lidar sensor (2.1) to transform the real pattern (M r ) is emitted to a vehicle environment or a robot environment within at least a portion of the acquisition area of ​​the camera (3), and - Using the camera (3) to capture the real pattern (M r ), Its characteristics are: - using the laser radar sensor (2.1), a virtual pattern (M) generated in the coordinate system of the laser radar sensor (2.1) is v ) is projected onto a virtual plane (E) in the vehicle environment or robot environment v )superior, - by means of the laser beam emitted by the laser radar sensor (2.1) passing through the virtual plane (E v ) and generates a virtual pattern (M) on a real projection surface (A) v ) is the associated real pattern (M r ), - The real pattern (M) captured by the camera (3) r ) is calculated based on the surface orientation of the real projection surface (A) to the virtual plane (E v ), thereby generating a corrected virtual pattern (M) in the coordinate system of the camera (3) ev ),and - By v ) and the corrected virtual pattern (M ev ) to calibrate the camera (3) and / or the lidar sensor (2.1) by comparison.

2. The method according to claim 1, characterized in that: The surface direction of the real projection surface (A) is based on the distance from the real pattern (M) determined by means of the laser radar sensor (2.1). r ) is determined by the distance of the image point.

3. The method according to claim 1 or 2, wherein For this virtual pattern (M v ) and the corrected virtual pattern (M ev ), determine at least one transformation formula for converting the virtual pattern (M v ) is converted to the corrected virtual pattern (M ev ) and / or for the corrected virtual pattern (M ev ) is converted to the virtual pattern (M v )middle.

4. The method according to claim 1 or 2, characterized in that: During calibration, an azimuth error and / or an elevation error of the camera (3) and / or the lidar sensor (2.1) is determined.

5. The method according to claim 1 or 2, characterized in that: In order to generate the virtual pattern (M v ), by means of a swivel mirror of the laser radar sensor (2.1), a laser beam emitted by the laser radar sensor (2.1) is deflected.

6. The method according to claim 1 or 2, wherein The real pattern (M) is captured by means of the camera (3) r ), the multiple images (B) successively captured by the camera (3) are integrated.

7. The method according to claim 1 or 2, characterized in that: For calibration, the camera (3) is switched to calibration mode.

8. The method according to claim 1 or 2, characterized in that: The virtual pattern (M) is generated by infrared laser radiation. v ).

9. The method according to claim 1 or 2, characterized in that: In order to generate the image (B) captured by means of the camera (3), the laser beam reaching the camera (3) is filtered by means of the camera's own infrared filter.

10. The method according to claim 9, characterized in that: - using an infrared filter such that the infrared laser beam emitted by the lidar sensor (2.1) and reflected by the real projection surface (A) can penetrate the infrared filter, and / or The infrared filter is switched in the calibration mode of the camera (3) so that the infrared laser beam emitted by the lidar sensor (2.1) and reflected by the real projection surface (A) can pass through.

Images