Method for the position calibration of a camera and a lidar sensor using a calibration board and a position calibration system

Description

[0001] The invention relates to a method for position calibration of a camera and a lidar sensor using a calibration board, as well as a position calibration system for position calibration of the camera and the lidar sensor.

[0002] Among other things, various types of obstacle detection systems are used in autonomous vehicles for obstacle detection. For example, video cameras and lidar sensors are used to detect obstacles. The images generated by the obstacle detection systems are preferably evaluated together. It is important to know where a detected obstacle is located in an image taken by one obstacle detection system and in another image taken by the other obstacle detection system. The aim is to be able to merge the results of the obstacle detection systems. However, mechanical tolerances during the installation of the obstacle detection systems, as is the case during vehicle assembly, are problematic. A small error in the alignment of the lidar sensor causes a large deviation at a distance of just 50 m.An obstacle detected in the lidar sensor's image is suddenly in a completely different location. A detected collision is suddenly no longer a collision, and an obstacle detected outside the roadway can suddenly become a dangerous situation.

[0003] CN 110 766 758 B describes the use of a standard camera and a depth sensor in combination. Furthermore, a calibration chart is shown, which includes known patterns, such as ARUCO codes, which are captured by the camera, and areas of higher reflectivity, which are captured by the depth sensor. Based on the resulting images, the calibration parameters between the camera and the depth sensor are determined.

[0004] US 2020 / 174107 A1 describes a method for acquiring image data of calibration objects in the environment using a plurality of image sensors. Furthermore, a three-dimensional LiDAR point cloud is acquired based on LiDAR data by a LiDAR sensor. Subsequently, a three-dimensional image point cloud is generated based on the image data and the three-dimensional LiDAR point cloud.

[0005] US 2021 / 316669 A1 describes a polyhedral calibration target for vehicles used to calibrate multiple sensor types, such as cameras and LIDAR. During calibration, visual and distance measurements of a polyhedral target are acquired to calibrate the sensors by mapping detected points and surfaces.

[0006] US 2015 / 352859 A1 describes a two-dimensional (2D) scanning system that uses a rapidly rotating raster polygon mirror to generate straight scan lines on an image surface. To minimize distortion, the angle of incidence of the laser beam on the raster polygon mirror and the tilt angle of the mirror's rotation axis are optimized. The system uses specially configured scanning optics that generate distortion to compensate for the distortion caused by the polygon mirror. This allows the system to generate distortion-free two-dimensional scans with a single rotating element, which is useful for applications such as laser phosphor displays or laser printers.

[0007] From Chai, Z., Sun, Y., and Xiong, Z. (2018). "A novel method for lidar camera calibration by plane fitting." 2018 IEEE / ASME International Conference on Advanced Intelligent Mechatronics (AIM), IEEE, 286-291, it is known that a three-dimensional calibration cube is used. From the perspective of the lidar sensor, up to three orthogonal surfaces are then determined in 3D space. From this, the overall position of the cube in 3D space can be determined. Each partial surface is provided with a marker code, so that the position of the code from the camera's perspective in 3D can be deduced by backprojection. These separately determined values ​​are then used to determine the registration between the 3D space from the perspective of the lidar sensor and the 3D space from the camera's perspective.

[0008] The disadvantage of this method is that the camera must already be calibrated. Furthermore, the calibration cube is large and unwieldy, which makes it difficult to use in practice.

[0009] It is therefore the object of the present invention to provide a method for position calibration of a camera and a lidar sensor, which can be carried out particularly efficiently.

[0010] The object is achieved by the method for position calibration of a camera and a lidar sensor with a calibration board according to independent claim 1, as well as by the position calibration system according to claim 13. Claims 2 to 12 describe advantageous developments of the method, and claim 14 describes an autonomously driving vehicle comprising a position-calibrated camera and lidar sensor.

[0011] The following describes the method according to the invention for the position calibration of a camera and a lidar sensor using a calibration chart. The calibration chart comprises known patterns. These known patterns exhibit varying brightness. Preferably, the known pattern comprises the colors black and white. Furthermore, additional reflection areas are applied to the calibration chart, which exhibit higher reflectivity than the known patterns of varying brightness. Reflectivity is the ratio between reflected and incident intensity as an energy quantity, for example, in the case of electromagnetic waves (luminous flux).

[0012] In a first step, at least one image of the calibration plate is captured with the camera. The pose of the calibration plate relative to the camera is determined based on known patterns of varying brightness. The term "pose" refers to the position and orientation of the calibration plate, particularly in the camera coordinate system.

[0013] In a second process step, a laser beam is emitted from the lidar sensor onto the calibration plate. This specifically means that the laser beam is emitted from the lidar sensor not only onto the calibration plate, but also onto the areas surrounding the calibration plate.

[0014] In a third process step, at least one image is taken with the lidar sensor and the areas with high reflectivity are determined based on intensity values ​​of the laser light reflected from the calibration plate.

[0015] In a fourth process step, the pose of the calibration plate relative to the lidar sensor is determined based on the known reflection areas and the identified areas of high reflectivity.

[0016] In principle, the image captured by the camera and the image captured by the lidar sensor can be recorded in parallel. Therefore, the process steps do not necessarily follow a chronological sequence. Of course, the laser light must be emitted by the lidar sensor first before the pose of the calibration plate relative to the lidar sensor can be determined. However, the determination of the pose of the calibration plate relative to the camera can be performed in parallel.

[0017] In a first alternative, both images can then be converted into a common coordinate system (CCS) based on the determined poses. In this case, the images captured by the camera and the lidar sensor are converted into a common coordinate system (CCS).

[0018] In a second alternative, an image captured by one sensor is converted into the coordinate system of the other image captured by the other sensor based on the determined poses. This way, the image captured by the camera can be converted into the coordinate system of the image captured by the lidar sensor. In this case, a transformation from the camera coordinate system (CCS) to the lidar coordinate system (LCS) takes place. Conversely, the image captured by the lidar sensor can also be converted into the coordinate system of the image captured by the camera. In this case, a transformation from the lidar coordinate system (LCS) to the camera coordinate system (CCS) takes place.

[0019] In a third alternative, conversion values ​​are determined based on the determined poses to convert future images into a common coordinate system (GCS), or to convert a future image from one sensor into the coordinate system of the other image and thus of the other sensor. Thus, using conversion values, a future image from the camera can be converted from the camera coordinate system (CCS) to the lidar coordinate system (LCS). Alternatively, using conversion values, a future image from the lidar sensor can be converted from the lidar coordinate system (LCS) to the camera coordinate system (CCS).

[0020] It is particularly advantageous that a single calibration table can be used to determine the poses of the calibration table relative to the camera and relative to the lidar sensor. This then makes it possible to convert images captured by the camera and / or the lidar sensor into the respective other coordinate system. If an object is detected in a camera image, it is possible to immediately determine the position in the lidar sensor image where this object must be located. The same applies to an object in the lidar sensor image. It is possible to immediately determine the position in the camera image where this object must be located.

[0021] It goes without saying that the calibration board can also simply comprise a painted wall. Therefore, the calibration board is not necessarily a separate structure that is placed in the room or hung on a wall.

[0022] Further advantages over prior art approaches include increased robustness and reduced calibration time, as required for industrial applications. By applying additional reflection areas to the calibration plate, highly characteristic intensity features in the image, i.e., in the lidar sensor's data, can be used to detect the calibration plate. This means that the calibration plate can be reliably detected in virtually any environment. However, with the purely geometric features described in the prior art, incorrect detections during position calibration can easily occur in any environment without prior knowledge.

[0023] Furthermore, alignment of the lidar sensor in the environment is easier. Diagnosing errors in the lidar sensor image, i.e., in the lidar data, is also easier. This makes it easier to determine why the data at a particular location does not look as expected.

[0024] Furthermore, extensive object information is made available. The spatial position of objects can be determined from the fused data (converted images from the various sensors). Additional properties can be determined from the camera images using object recognition or code and text reading. For example, it is possible to distinguish whether the object in the image is a person or not. Unique identification via a QR code is also possible. This could lead to more efficient anti-collision solutions; for example, a person must be avoided more slowly and carefully than a pallet. In addition or alternatively, documentation can be carried out during palletizing tasks, for example by simultaneously identifying goods loaded onto an automated guided vehicle. Furthermore, object detection can be improved in anti-collision solutions.In the image from the lidar sensor, i.e., the lidar data, a small object, such as the fork of a forklift, could be detected by only one or a few points. Using the camera images, i.e., the camera data, it could then be determined whether this is a disturbance or a relevant object with a potential collision risk.

[0025] In a further development of the position calibration method, the camera and the lidar sensor are arranged in a fixed position relative to each other. This means that the distance and orientation between the camera and the lidar sensor remain the same, i.e., constant. However, it is possible for the camera and lidar sensor to be moved together. For example, they can be mounted together on a vehicle that is moving autonomously. The distance and orientation of the camera and the lidar sensor then correspond to the distance and orientation between the camera and the lidar sensor during subsequent operation.

[0026] In a further development of the position calibration method, the camera and the lidar sensor record the respective image simultaneously. This means that the same scene is recorded. The time offset between the recording of the various images is preferably less than 1 second, 500 ms, 200 ms, 100 ms, 50 ms, 20 ms, 10 ms, or less than 1 ms. To achieve a minimal time offset, the respective image from the camera and the lidar sensor can be provided with a timestamp, whereby the time information from the timestamp can be used to determine which images were recorded "simultaneously" by the lidar sensor and the camera. The smallest possible time offset is achieved by synchronizing the local timers of the lidar sensor and camera with an external timer or by providing the time signal from an external timer directly to the lidar sensor and the camera.By sorting the images from the lidar sensor and the camera based on their timestamps, the smallest possible time offset can be achieved.

[0027] In a further development of the position calibration method, the additional reflection areas comprise reflective strips. These are preferably glued to the calibration board.

[0028] In a further development of the position calibration method, the additional reflection areas, particularly in the form of the reflection strips, are aligned at an angle to each other. The reflection areas are not all aligned parallel to each other, because otherwise, it would not be possible to clearly identify the pose between the calibration plate and the lidar sensor using laser lines emitted by the lidar sensor in the form of laser light.

[0029] In a further development of the position calibration method, the known patterns include ChArUco patterns. Such patterns can be used for both intrinsic calibration, such as estimating and correcting the lens distortion, focal length, and / or focal point of the camera, and extrinsic calibration, such as determining the pose of the calibration plate in the camera coordinate system. This method was described, for example, by Garrido-Jurado, S., Mu-Salinas, R., Madrid-Cuevas, F., and Mar Jim z, M. (2014). "Automatic generation and detection of highly reliable fiducial markers under occlusion." Pattern Recogn. 47, 6 (June 2014), 2280-2292. The open source library "opencv camera calibration (2022)" can be used for this purpose. Opencv camera calibration (2022). "Camera calibration with opencv,<https: / / docs.opencv.org / 4.x / d4 / d94 / tutorial_camera_calibration.html> .

[0030] Intrinsic camera calibration is determined directly from the camera images. This significantly simplifies practical use, as the camera does not need to be calibrated beforehand. This means that cameras from other manufacturers, for example, can easily be integrated into the process. Prior camera calibration always carries the risk that the optical properties of the camera system may have changed between the camera's intrinsic calibration and the registration of the lidar sensor against the camera (e.g., due to vibration, focus adjustment, and thermal expansion). Since the calibration plate is small compared to typical measurement rooms, it could also remain permanently in the measurement room to regularly check the measurement parameters by re-imaging the calibration plate with the lidar sensor and camera (quality assurance).

[0031] It is particularly advantageous that a single calibration table is used to determine all calibration steps (camera: intrinsic and extrinsic, lidar sensor: extrinsic). This creates a direct relationship between the laser-related measurement areas (reflection areas) and the camera-related measurement areas (ChArUco pattern) directly via the calibration table. This increases the robustness of the method and avoids setup errors compared to other methods in which intrinsic and extrinsic calibration are performed separately. Finally, the spatially compact design of the calibration table makes it suitable for use in industrial environments where available space may be limited or some areas are difficult to access, making the construction of complex calibration fixtures impossible.

[0032] In a further development of the position calibration method, the known patterns are located on the calibration board in one step. In a further step, the camera's intrinsic parameters, in particular focal length, focal point, and / or lens distortion, are determined using the known patterns. Subsequently, the camera is calibrated using the intrinsic parameters. Therefore, the camera does not need to be pre-calibrated. This greatly simplifies practical use.

[0033] In a further development of the method for position calibration, a large number of images are used to determine the intrinsic parameters of the camera. Images in which the projection error exceeds a threshold value are discarded. To determine the intrinsic parameters, the camera takes images of the same scene. The relative position between the camera and the calibration plate is preferably unchanged. However, it is also conceivable that the position or orientation of the calibration plate and / or the camera has changed in at least two of the images or in all images. Preferably, a certain number of images are also taken with the camera, whereby this number lies above a further threshold value. Further preferably, at least 2, 3, 6, 8, 12, 16, 20 or more than 25 images are used.

[0034] In a further development of the position calibration method, the calibration plate is positioned such that it lies entirely within the camera image and is intersected by a plurality of laser lines, i.e., scan lines of the lidar sensor. In principle, it is conceivable that the method is not limited to a lidar sensor that generates a specific number of laser lines, such as four. It thus also allows the use of lidar sensors with diverse point distribution characteristics, such as single-plane lidar sensors or lidar sensors with pseudo-random point distributions.

[0035] According to the invention, the lidar sensor generates more than 2, 3, 4, or more than 5 laser lines. These laser lines preferably run parallel to each other and are preferably generated simultaneously. This allows for a very precise determination of the pose, i.e., the position and orientation of the calibration plate relative to the lidar sensor, because different laser lines cross different reflection areas. Especially in the case where the reflection areas are aligned at an angle to each other, the pose of the calibration plate relative to the lidar sensor can be determined very precisely.

[0036] According to the invention, a curvature of the laser lines is compensated for by a known deviation from a straight laser line. Strictly speaking, the laser line projected by the lidar sensor onto the calibration chart is not a straight line, but rather a flatly curved hyperbola. The deviation from a straight line is not particularly large, but it is still possible to account for the curvature of the hyperbola. This is achieved by deriving the deviation from a straight laser line from a look-up table. The look-up table is calculated and saved once for each laser line.

[0037] In a further development of the position calibration method, a process step is used to determine which sections of the reflected laser lines have an intensity profile above a threshold value and which sections of the reflected laser lines lie in one plane. Sections above the threshold value are areas of high reflectivity. This is how the intensity profile is binarized. All additional reflection areas on the calibration board lie in one plane. Such an image must also result from the reflected laser lines. Points in space are determined for the reflected laser lines (e.g. X, Y, Z), and a corresponding intensity value is determined for each point. Furthermore, the reflectivity of the additional reflection areas is higher than the surrounding area.In a further process step, the sections thus determined are compared with the known additional reflection areas. The location of the additional reflection areas on the calibration plate is known, or has been determined from at least one camera image. This allows the calibration plate to be detected in the image from the lidar sensor.

[0038] In a further development of the position calibration method, the adjustment is performed by determining a displacement and a rotation angle of the determined sections relative to the origin of the calibration table in order to determine the pose of the calibration table relative to the lidar sensor. For example, the reflected laser lines can be shifted in the plane until the areas of the reflected laser lines whose intensity values ​​are above a threshold correspond to the areas on the calibration table where the additional reflection areas are located. This can be done using an iterative process, which also involves rotating the reflected laser lines.

[0039] In a further development of the position calibration method, data from at least one radar sensor could also be included. In this case, the calibration panel could also include radar reflectors. These radar reflectors can be active or passive radar reflectors.

[0040] In a further development of the position calibration method, preferably multiple images are created by the camera and / or the lidar sensor. Preferably, the camera and the lidar sensor are stationary. The position and alignment between the camera and the lidar sensor are also fixed and do not change. The calibration plate can then be moved while various images are captured by the camera and the lidar sensor. Alternatively, the calibration plate can be stationary. In this case, a combination of camera and lidar sensor can move, while the position and alignment between the camera and the lidar sensor remain unchanged. As a result, the alignment between the calibration plate and the camera / lidar sensor differs in the individual images.

[0041] In a further development of the position calibration procedure, additional test calibration boards can be set up. These test calibration boards can be used to verify the position calibration.

[0042] In a further development of the position calibration method, the lidar sensor can have an azimuth detection field of more than 120°, 180°, or more than 240°. The azimuth detection field can also be 360°. Furthermore, at least one additional camera and at least one additional calibration board are used. More than one additional camera could also be used. In this case, all cameras could be compared against the lidar sensor. The method would then allow all images to be converted into a common coordinate system based on the determined poses. It would also be conceivable for the images from at least two, but not all, sensors to be converted into the coordinate system of the remaining sensor (e.g., the lidar sensor). In principle, it would also be possible to determine conversion values ​​in this case in order to convert future images into a common coordinate system.It would also be possible to determine conversion values ​​to convert a future image into the coordinate system of another future image from a different sensor.

[0043] In a further development of the position calibration method, the pose of the calibration board relative to the lidar sensor could also be optimized using additional prior knowledge. A Monte Carlo search can be used to determine the desired pose. The search space can then be significantly restricted based on the pose of the calibration board relative to the camera determined during the extrinsic camera calibration. This assumes that the lidar sensor and camera are close to each other, or that their relative positions are roughly known. However, this is not a mandatory restriction and merely reduces the search effort in the parameter space. The remaining free parameters are essentially limited to the deviation angle from the horizontal of at least one laser line sweeping across the calibration board and its height on the calibration board. A distance measure, such asthe Hamming distance, which indicates the distance between the recorded intensity values ​​of the reflected laser light that are above a threshold and the position of the additional reflection areas predicted using the current transformation.

[0044] In a further development of the method, the position and / or orientation of the calibration plate relative to a camera and lidar sensor assembly can be changed. Preferably, the calibration plate is moved around the camera and lidar sensor assembly. However, the camera and lidar sensor assembly could also be moved around the calibration plate. The distance and orientation between the camera and lidar sensor remain unchanged. The camera and lidar sensor then acquire images again, and the pose of the calibration plate relative to the camera and lidar sensor is determined again. This allows, for example, more precise conversion values ​​to be determined.

[0045] The position calibration system according to the invention comprises a camera, a lidar sensor, a calibration board, and a processing unit. The calibration board has known patterns, wherein the known patterns comprise different brightnesses. Additional reflective regions which have a higher reflectivity than the known patterns of different brightnesses are applied to the calibration board. The camera is designed to record an image of the calibration board, wherein the processing unit is designed to determine a pose of the calibration board relative to the camera based on the known patterns. These known patterns are captured in the camera image. The lidar sensor is designed to emit laser light onto the calibration board, wherein the laser light comprises more than two, three, four, or more than five laser lines which are arranged at a distance from one another.The lidar sensor is further configured to capture an image, wherein the processing unit is configured to determine areas of high reflectivity based on intensity values ​​of the laser light reflected from the calibration plate. The processing unit is configured to compensate for a curvature of the laser lines by a known deviation from a straight line. The deviation of the laser line projected onto the calibration plate from a straight laser line is taken from a look-up table that is calculated and stored once for each laser line. The processing unit is configured to convert both images into a common coordinate system (GCS) based on the determined poses.Alternatively, the processing unit is designed to convert an image into the coordinate system of the other image based on the determined poses. Alternatively, the processing unit is designed to determine conversion values ​​based on the determined poses in order to convert future images into a common coordinate system (GCS) or to convert a future image from one sensor (camera or lidar sensor) into the coordinate system of another future image from the other sensor (lidar sensor or camera). Provision can be made to transform the lidar data into a 2D image coordinate system. The position of at least some pixels in the 2D image coordinate system could also be transformed into the 3D system.

[0046] Furthermore, an autonomously driving vehicle, in particular in the form of a forklift truck, is specified. The autonomously driving vehicle comprises a camera, a lidar sensor and a processing unit. The camera and the lidar sensor are calibrated to one another according to one of the preceding claims and are arranged on the vehicle so as to be relatively immovable relative to one another. The lidar sensor is designed to continuously monitor the roadway, in particular in front of the vehicle, for obstacles. The processing unit is designed to detect at least one object in the data (images) of the lidar sensor, which, for example,is smaller than a threshold value, with the camera data in order to brake the vehicle and / or initiate a steering movement to avoid a collision and / or issue a warning if the object poses a risk of collision, or not to intervene in the vehicle's movement and / or issue a warning if the object does not pose a risk of collision. The warning can be visual and / or acoustic and / or haptic, e.g., through vibrations.

[0047] The invention is described below purely by way of example with reference to the drawings. They show: Figure 1: an embodiment with a camera and a lidar sensor monitoring a spatial area in a production environment; Figures 2, 3, 4: a position calibration system with the camera, the lidar sensor and a calibration board; Figure 5: a path of laser light reflected from the calibration board, with different sections of the laser light having different intensity values; Figure 6: a possibility describing how the sections from Figure 5be adjusted with additional reflection areas on the calibration board; Figure 7: an image of a lidar sensor in which the calibration board is located; Figures 8, 9: embodiments that explain that the laser line of the lidar sensor is not a straight line, but a flatly curved hyperbola; Figures 10A, 10B, 10C: embodiments that explain how the pose of the calibration board relative to the lidar sensor is determined; and Figure 11: a flowchart describing a method for position calibration of a camera and a lidar sensor using a calibration board.

[0048] Figure 1shows the use of a camera 1 and a lidar sensor 2, which jointly monitor a spatial area 3 of a production environment. In this case, a robot 4 is shown in the production environment. To prevent people from inadvertently entering the movement area of ​​the robot 4, the spatial area 3 in which the robot 4 is located is monitored with the camera 1 and the lidar sensor 2. In this embodiment, the camera 1 and the lidar sensor 2 are arranged in a common housing 5. However, they can also be housed in different housings. Both the camera 1 and the lidar sensor 2 have a monitoring field 6. The monitoring field 6 of the camera 1 is shown with a solid line, whereas the monitoring field 6 of the lidar sensor 2 is shown with a dotted line. Both monitoring fields 6 are aligned so that they overlap with one another, at least in part.As will be explained later, the data from camera 1 and the data from lidar sensor 2 can be fused, allowing immediate determination of the location of a detected object 7 in the data from camera 1 and the data from lidar sensor 2, and vice versa. This enables significantly more efficient monitoring of the spatial area 3.

[0049] This fusion can be performed with a processing unit 8, which is arranged, for example, in the common housing 5. In this case, the common housing 5 is mounted on a wall 9.

[0050] Camera 1 can be a photo camera or a video camera.

[0051] In the Figures 2 , 3 , 4A position calibration system 10 is shown with the camera 1, the lidar sensor 2, and a calibration table 11. The calibration table 11 comprises known patterns 12 of varying brightness. The known patterns 12 are preferably ChArUco patterns. These comprise a checkerboard structure with ArUco patterns. The patterns 12 themselves are known in terms of size and orientation relative to one another.

[0052] Camera 1 and lidar sensor 2 are fixed, meaning they cannot be moved relative to each other. This arrangement is maintained after position calibration, i.e., when used in an autonomous vehicle.

[0053] In Figure 3It is shown that the camera 1 takes an image 13 in which the calibration board 11 is located. Using the processing unit 8, it is possible to determine a pose of the calibration board 11 relative to the camera 1. This is done by detecting the known patterns 12. The determination of the pose (position and orientation) of the calibration board 11 relative to the camera 1 is also referred to as extrinsic camera calibration. Furthermore, an intrinsic camera calibration can also be performed based on the recorded image 13. In particular, the focal length, focal point and / or lens distortion are determined. This works based on the detected known patterns 12. The camera 1 therefore does not have to be pre-calibrated.

[0054] Preferably, the setup consisting of camera 1 and lidar sensor 2 is moved around the calibration plate 11 so that several images 13 are captured by the camera 1 from different angles onto the calibration plate 11. Of course, the calibration plate 11 could also be moved in addition or as an alternative.

[0055] In the Figures 2 , 3 and 4 It is also shown that the calibration panel 11 comprises additional reflective regions 14 that have a higher reflectivity than the known patterns 12. These additional reflective regions 14 comprise, in particular, a plurality of reflective strips. The reflective strips are at least partially aligned at an angle to one another. The known patterns 12 can be partially covered, for example, pasted over, by the additional reflective regions 14.

[0056] In Figure 4It is shown that the lidar sensor 2 records an image 15 in which the calibration plate 11 is located. This is done by emitting laser light 16, in particular in the form of several laser lines 17. The reflected laser light 16 in the form of the laser lines 17 is visible in the recorded image 15 of the lidar sensor 2. In the recorded image 15 of the lidar sensor 2, areas 18 (see Figure 5 ) with high reflectivity. This is done based on intensity values ​​of the laser light 16 reflected by the calibration panel 11. The processing unit 8 is designed to determine a pose of the calibration panel 11 relative to the lidar sensor 2 based on the known reflection areas 14 and the determined areas 18 with high reflectivity.

[0057] Preferably, the setup consisting of camera 1 and lidar sensor 2 is moved around the calibration plate 11 so that several images 15 are captured by the lidar sensor 2 from different angles onto the calibration plate 11. Of course, the calibration plate 11 could also be moved in addition or as an alternative.

[0058] The camera 1 and the lidar sensor 2 preferably take images 13, 15 of the calibration board 11 synchronously with each other.

[0059] Preferably, several laser lines 17 of the lidar sensor 2 cross the calibration plate 11 and thus the additional reflection areas 14. Because the additional reflection areas 14 comprise reflection strips aligned at an angle to one another, the reflected laser light 16 can be used to precisely determine which part of the calibration plate 11 was crossed by the laser lines 17.

[0060] The lidar sensor 2 preferably emits more than two, three, four, or more than five laser lines 17, which are preferably arranged parallel to one another. The laser lines 17 are preferably arranged completely non-overlapping with one another.

[0061] The processing unit 8 is further configured to convert both images 13, 15 into a common coordinate system (CCS) based on the determined poses. In this case, the exact location of an object 7 can be determined.

[0062] Alternatively, the processing unit 8 is configured to convert an image 13, 15 into the coordinate system of the other image 15, 13 based on the determined poses. It is then also possible to determine in which area of ​​the other image 15, 13 the object 7 must be located.

[0063] Additionally or alternatively, the processing unit 8 is particularly configured to determine conversion values ​​based on the determined poses in order to use the conversion values ​​to convert future images 13, 15 into a common coordinate system (GCS). Using the conversion values, a future image 13, 15 can also be converted into the coordinate system of the other image 15, 13. This enables efficient fusion of both images 13, 15, and detected objects 7 in one image 13, 15 can be verified using the other image 15, 13.

[0064] Figure 5shows a captured image 15 of the lidar sensor 2. The image 15 contains the reflected laser light 16 with the laser lines 17 emitted by the lidar sensor 2. The image 15 contains areas of high reflectivity 18, i.e., areas of high light intensity. These areas of high reflectivity 18 arise when the laser light 16, i.e., the laser lines 17 emitted by the lidar sensor 2, strike the additional reflection areas 14 of the calibration plate 11.

[0065] Furthermore, areas with low reflectivity 19 are shown. In these areas, the intensity of the reflected laser light 16 is lower than in the areas with high reflectivity 18. The laser lines 17 emitted by the lidar sensor 2 strike areas of the calibration panel 11 that absorb a higher light output.

[0066] In Figure 6This is illustrated using another exemplary embodiment. The additional reflection areas 14 are drawn on the calibration plate 11. The known patterns 12 have been omitted in this case for clarity. Furthermore, the reflected laser light 16 is shown in the form of laser lines 17. The reflected laser light 16 has a varying intensity profile depending on which areas of the calibration plate the laser light 16 has struck. Areas with high reflectivity 18, i.e., a high intensity value, are shown brighter than areas with low reflectivity 19.The processing unit 8 can analyze the image 15 of the lidar sensor 2 to determine the area in which the calibration plate 11 is located and how the calibration plate 11 must be aligned relative to the lidar sensor 2 so that the detected areas of high reflectivity 18 match the known additional reflection areas 14 of the calibration plate 11. This will be explained in more detail below.

[0067] Figure 7shows an image 15 of the lidar sensor 2 in which the calibration table 11 is located. The method described below is intended to illustrate only one possible example. The measurement data of the lidar sensor 2 are preferably transformed into Cartesian coordinates so that the 3D position in lidar coordinates and its intensity value are known for each measurement point. The image 15 of the lidar sensor 2 therefore comprises a large number of measurement points. The set of all measurement points can also be referred to as a point cloud. The calibration table 11 is now iteratively located in the point cloud. The measurement point P max with the highest intensity value is searched for. It is examined whether this point lies within a plane. The detection of such a plane in the point cloud is carried out using known methods, e.g. by segmentation pcl segmentation (2022) of the PointCloudLibrary (PCL) pcl (2022).Starting from the selected measurement point P max, this segmentation yields all measurement points that lie in a plane with a specified maximum distance. If the measurement point P max lies in a flat surface with the dimensions of the calibration plate 11, then the calibration plate is found. Otherwise, the described procedure is repeated with the measurement point with the next lowest intensity value. Optionally, the search area can be restricted by specifying a 3D cuboid to avoid errors caused by reflections or artifacts.

[0068] The intensity values ​​of the measurement points lying on the flat surface are then binarized. The binarization is performed using an intensity histogram. The binarization threshold is determined such that the percentage of bright points (intensities above the threshold) corresponds to the reflective portion on the calibration panel 11. The ratio of reflective to non-reflective points on the calibration panel 11 is known and constant (width of the reflective stripes of the additional reflective areas 14 relative to the width of the calibration panel 11). Such a binarization of the lidar point cloud on the calibration panel is, for example, Figure 5 shown.

[0069] The binarized intensity values ​​of the reflected laser light 16 in the form of laser lines 17 are compared against the calibration pattern, i.e., against the known additional reflection areas 14 on the calibration plate 11, and the intersection lines of the laser lines 17 on the calibration plate 11 are thereby determined. This determines the position and rotation angle of the laser lines 17 relative to the origin of the calibration plate 11. The determination is performed, for example, by iteration over all permissible positions and / or by a search method such as a Monte Carlo search. The iterative approach varies the start and end points of the intersection lines and includes, for example, the following options: a) For each intersection line, the Hamming distance between the binarized intensity values ​​and the points on the binarized reference chart is calculated. The binarized reference chart is a distortion-free and rectified sample of the calibration chart 11, which only distinguishes reflective points (additional reflection areas 14) from non-reflective points (remaining areas). b) The intersection line with the smallest Hamming distance determines the position and rotation angle of the laser lines 17 relative to the origin of the calibration chart 11.

[0070] Also in Figure 6 A curve with binarized intensity values ​​is shown, which are mapped to a preferentially binarized reference pattern. It can be seen how the best match was determined by shifting and rotating the laser lines 17 in the reference pattern.

[0071] It should also be noted that the lidar sensor 2 does not scan the space in a plane, but rather through a flat cone. The laser line 17 projected by the lidar sensor 2 onto the calibration plate 11 is therefore not a straight line, but a flatly curved hyperbola.

[0072] The hyperbolic projection of the laser line 17 and its difference to the straight line dz B (y B ) is in Figure 8 shown. Here is: the laser line 17 in 3D lidar coordinates L scan =(x L , y L , z L ) in [meters], its projection onto the 2D table coordinates L proj =(y B , z B ) in [meters] and the cone x 2< L +y 2< L =R 2< z 2< L for a given width w of the calibration panel 11. The parameter R parametrically describes the opening angle of the cone.

[0073] The Figures 10A, 10B and 10Cshow the iteration over a curved laser line by shifting the start and end points. At the end of the iteration, the regions of high reflectivity 18 of the reflected laser light 16, in the form of laser lines 17, lie exclusively or more than 90%, 95%, or more than 98% above the additional reflection regions 14.

[0074] Preferably, the curvature of the hyperbola is taken into account by extracting the deviation from the straight laser line 17 from a lookup table according to the invention. According to the invention, the lookup table is calculated and stored once for each laser line 17. Figure 9shows the deviation of the hyperbolas from the straight laser line 17, which is stored in the lookup table. The width of the graph corresponds approximately to the width of the calibration table 11. The figure is exaggerated in the Y direction. Even if the deviations between the intersection hyperbola and an approximate intersection line are small, taking the hyperbolic intersection into account increases the accuracy of the image. The difference dz B (y B ) from the straight laser line 17 is stored in the lookup table for each value y B . The values ​​dz B (y B ) can be calculated as follows: x L = x L 1 + y B px w px x L 2 − x L 1 y L = y L 1 + y B px w px y L 2 − y L 1 z L 1 = 1 R x L 1 2 + y L 1 2 z L 2 = 1 R x L 2 2 + y L 2 2 z L = 1 R x L 2 + y L 2 dz B y B = z L − y B px w px z L 2 − z L 1 + z L 1

[0075] Now the following is known: the position and orientation (pose) of the calibration plate 11 in camera coordinates; the measurement points of the laser lines 17 in lidar coordinates; and the position of the laser lines 17 relative to the origin of the calibration plate 11 and thus the pose of the calibration plate 11 relative to the lidar sensor 2.

[0076] This means that for every measurement point P Lidar in 3D lidar coordinates that is located on the calibration board, a corresponding point in 3D camera coordinates can be specified. A transformation T Lidar→Board can be specified with which a measurement point P Lidar =(X Lidar , y Lidar , Z Lidar ) can be mapped to a point on the calibration board 11 P Board =(y Board , z Board ) (in the board reference system). Since the pose of the calibration board 11 in the camera coordinate system is also known, a further transformation T Board→Camera can be calculated. This means that the point P Board =(y Board , Z Board ) can be mapped to a measurement point P Camera =(X Camera , y Camera , z Camera ) in the camera coordinate system. From these point correspondences, a transformation T Li-dar→camera can be estimated using known methods (Least Mean Square), which maps each measurement point in lidar coordinates into a 3D point in camera coordinates (ienot just the points on the calibration table). The 3D transformation can be estimated, for example, as described in opencv pose estimation (see opencv pose estimation (2022)). Using the determined intrinsic camera parameters, the 3D camera points can be transformed into 2D image points. The determination and application of this transformation T KameraIntrinsic is carried out, for example, using well-known methods such as in opencv projection (see opencv projection (2022)). By concatenating the transformations T Lidar→Kamera and T CameraIntrinsic, the measurement points of the lidar point cloud can now be transformed into the current image 13 of camera 1, and the desired assignment of 3D lidar points to 2D image points is possible.

[0077] Figure 11 shows a flowchart describing a method for position calibration of the camera 1 and the lidar sensor 2 with the calibration board 11.

[0078] In a first method step S 1, at least one image 13 of the calibration board 11 is taken with the camera 1 and a pose of the calibration board 11 relative to the camera 1 is determined based on the known patterns 12 on the calibration board 11.

[0079] In a second method step S 2, laser light 16, in particular in the form of laser lines 17, is radiated from the lidar sensor 2 onto the calibration board 11.

[0080] In a third method step S 3, an image 15 is taken of the calibration panel 11 using the lidar sensor 2, and areas with high reflectivity 18 are determined based on intensity values ​​of the laser light 16 reflected by the calibration panel 11, in particular in the form of the laser lines 17.

[0081] In a fourth method step S 4, a pose of the calibration plate 11 relative to the lidar sensor 2 is determined based on the known additional reflection areas 14 and the determined areas with high reflectivity 18.

[0082] In a fifth method step S 5, in a first alternative, both images 13, 15 can be converted into a common coordinate system (GKS) based on the determined poses. In a second alternative, one image 13, 15 can be converted into the coordinate system of the other image 15, 13 based on the determined poses. Thus, an image 13 of camera 1 can be converted into the lidar coordinate system (LKS) of the lidar sensor 2. Alternatively, an image 15 of the lidar sensor 2 can be converted into the camera coordinate system (KKS) of the camera 1. In a third alternative, conversion values ​​are determined based on the determined poses in order to convert future images 13, 15 into a common coordinate system (GKS) or to convert a future image 13, 15 into the coordinate system of another future image 15, 13.

[0083] The invention is not limited to the described embodiments. Within the scope of the invention, all described and / or illustrated features can be combined with one another in any way. In particular, it is clear that the transformation possibilities described are merely examples, and other mathematically possible solutions exist. List of reference symbols

[0084] camera 1 Lidar sensor 2 Room area 3 robot 4 Housing 5 Monitoring field 6 object 7 processing unit 8 Wall 9 Position calibration system 10 Calibration board 11 Known patterns 12 Image from camera 13 Additional reflection areas 14 Image of lidar sensor 15 Laser light 16 Laser lines 17 Areas with high reflectivity 18 Areas with low reflectivity 19 Procedural steps S1, S2, S3

Claims

1. A method for position calibration of a camera (1) and a LIDAR sensor (2) using a calibration board (11), wherein the calibration board (11) comprises known patterns (12), wherein the known patterns (12) have different brightnesses and wherein additional reflection regions (14) are present on the calibration board (11) that have a higher reflectivity than the known patterns (12) of different brightness, wherein the method comprises the following method steps: - recording (S1) at least one image (13) of the calibration board (11) with the camera (1) and determining a pose of the calibration board (11) relative to the camera (1) based on the known patterns (12); - transmitting (S2) a laser light (16) from the LIDAR sensor (2) to the calibration board (11), wherein the laser light (16) comprises more than two, three, four or more than five laser lines (17) which are arranged spaced apart from one another; - recording (S3) at least one image (15) with the LIDAR sensor (2) and determining regions of high reflectivity (18) based on intensity values of the laser light (16) reflected by the calibration board (11); - compensating a curvature of the laser lines (17) by a known deviation from a straight line, wherein the deviation of the laser line projected onto the calibration board from a straight laser line is taken from a lookup table which is calculated and stored once for each laser line; - determining (S4) a pose of the calibration board (11) relative to the LIDAR sensor (2) based on the known additional reflection regions (14) and the determined regions of high reflectivity (18); and the following method step (S5): a) converting both images (13, 15) into a common coordinate system based on the determined poses; or b) converting one image (13, 15) into the coordinate system of the other image (15, 13) based on the determined poses; and / or (c) determining conversion values based on the determined poses to convert future images (13, 15) into a common coordinate system or to convert a future image (13, 15) into the coordinate system of another future image (15, 13).

2. A method for position calibration according to claim 1, wherein the camera (1) and the LIDAR sensor (2) are arranged in fixed positions relative to one another.

3. A method for position calibration according to claim 1 or 2, wherein the camera (1) and the LIDAR sensor (2) simultaneously record the respective image (13, 15).

4. A method for position calibration according to any one of the preceding claims, wherein the additional reflection regions (14) comprise reflection strips.

5. A method for position calibration according to any one of the preceding claims, wherein the additional reflection regions (14), in particular the reflection strips, are oriented at an angle to one another.

6. A method for position calibration according to any one of the preceding claims, wherein the known patterns (12) comprise ChArUco patterns.

7. A method for position calibration according to any one of the preceding claims, wherein the method comprises the following method steps: - locating the known patterns (12) on the calibration board (11); - determining the intrinsic parameters, in particular the focal length, focal point and / or lens distortion of the camera (1) based on the known patterns (12); - calibrating the camera (1) by means of the intrinsic parameters of the camera (1).

8. A method for position calibration according to claim 7, wherein the method comprises the following method step: - using a plurality of images (13) to determine the intrinsic parameters of the camera (1), wherein those images (13) are discarded in which the projection error exceeds a threshold value.

9. A method for position calibration according to any one of the preceding claims, wherein the method comprises the following method step: - placing the calibration board (11) such that it lies completely in the image of the camera (1) and such that it is crossed by a plurality of scan lines, in particular laser lines (17), of the LIDAR sensor (2).

10. A method for position calibration according to any one of the preceding claims, wherein the method comprises the following method steps: - determining which sections (18) of the reflected laser lines (17): a) have an intensity development which lies above a threshold value; b) lie in one plane; - comparing the determined sections (18) with respect to the known additional reflection regions (14).

11. A method for position calibration according to claim 10, wherein the method step comparing comprises the following sub-method step: - determining a displacement and a rotation angle of the determined sections (18) relative to the origin of the calibration board (11) in order thus to determine the pose of the calibration board (11) relative to the LIDAR sensor (2).

12. A method for position calibration according to any one of the preceding claims, wherein the method comprises the following method steps: - changing the position and / or the alignment of the calibration board (11) relative to an assembly of the camera (1) and the LIDAR sensor (2) and repeating at least the method steps recording (S1), transmitting (S2), recording (S3) and determining (S4).

13. A position calibration system (10) comprising a camera (1), a LIDAR sensor (2), a calibration board (11), and a processing unit (8), wherein the calibration board (11) comprises known patterns (12), wherein the known patterns (12) have different brightnesses and wherein additional reflection regions (14) are present on the calibration board (11) that have a higher reflectivity than the known patterns (12) of different brightness, and wherein: - the camera (1) is configured to record at least one image (13) of the calibration board (11) and wherein the processing unit (8) is configured to determine a pose of the calibration board (11) relative to the camera (1) based on the known patterns (12); - the LIDAR sensor (2) is configured to transmit a laser light (16) to the calibration board (11), wherein the laser light (16) comprises more than two, three, four or more than five laser lines (17) which are arranged spaced apart from one another; - the LIDAR sensor (2) is configured to record at least one image (15) and wherein the processing unit (8) is configured to determine regions of high reflectivity (18) based on intensity values of the laser light (16) reflected by the calibration board (11); - the processing unit (8) is configured to compensate a curvature of the laser lines (17) by a known deviation from a straight line, wherein the deviation of the laser line projected onto the calibration board from a straight laser line is taken from a lookup table which is calculated and stored once for each laser line, and to determine the pose of the calibration board (11) relative to the LIDAR sensor (2) based on the known additional reflection regions (14) and the determined regions of high reflectivity (18); and wherein the processing unit (8) is configured: a) to convert both images (13, 15) into a common coordinate system based on the determined poses; or b) to convert one image (13, 15) into the coordinate system of the other image (15, 13) based on the determined poses; or c) to determine conversion values based on the determined poses to convert future images (13, 15) into a common coordinate system or to convert a future image (13, 15) into the coordinate system of another future image (15, 13).

14. An autonomously driving vehicle, in particular in the form of a forklift truck, comprising a camera (1), a LIDAR sensor (2), and a processing unit (8), wherein the camera (1) and the LIDAR sensor (2) are calibrated with respect to one another in accordance with any one of the claims 1 to 12, wherein the LIDAR sensor (2) is configured to continuously monitor the road in front of the vehicle for obstacles, wherein the processing unit (8) is configured to compare at least one object (7) in the data of the LIDAR sensor (2) that is smaller than a threshold value with the data of the camera (1) in order, in the event that: a) the object (7) poses a risk of collision, to brake the vehicle and / or to initiate a steering movement and / or to issue a warning; b) the object (7) does not pose a risk of collision, not to intervene in the driving movement of the vehicle and / or not to issue a warning.