Infrastructure multi-sensor device, method for calibration, and method for detecting an object in the surroundings

Abstract

The invention relates to a method (56) for calibrating an infrastructure multi-sensor device (14) which acquires surroundings data of an infrastructure device (10) and comprises at least a first sensor (24) and a second sensor (26) which are each intended for detecting at least a first object (28) in the surroundings of the infrastructure device (10), wherein the first sensor (24) is calibrated via sensor calibration (58), in that at least a first object parameter (38) of the first object (28) in the surroundings is measured by the first sensor (24) and is associated with a first sensor reference system (40) as a first allocation (42), and the first object parameter (38) is measured by the second sensor (26) and is associated with a second sensor reference system (50) as a second allocation (52), and subsequently the first allocation and the second allocation (42, 52) are compared, and the first sensor (24) is calibrated on the basis of this comparison of allocations (54). Furthermore, the invention relates to a method (56) for detecting an object (28) in the surroundings and to an infrastructure multi-sensor device (14).

Description

[0001] Infrastructure multi-sensor device, method for calibration and method for detecting an environmental object

[0002] The invention relates to a calibration method according to claim 1. Furthermore, the invention relates to a method for detecting an environmental object and an infrastructure multiple sensor device.

[0003] State of the art

[0004] DE 102016224 766 A1 describes a method for calibrating sensors on a mobile platform relative to one another. For this purpose, defined markers in the environment of the mobile platform are detected by the sensors. This procedure allows an extrinsic calibration of the sensors relative to one another, with each sensor detecting at least one defined marker. Based on the detected markers, the position and orientation of a first sensor in relation to the position and orientation of a second sensor can be calculated. The information encoded in the markers is used to determine the position and orientation of the sensors, whereby this determination is preferably carried out by comparing this information with a digital map.Starting from the determination of the position and orientation of the sensors, the extrinsic calibration of a sensor in relation to another sensor is carried out.

[0005] Disclosure of the invention

[0006] According to the present invention, a calibration method having the features of claim 1 is proposed.

[0007] This allows sensor calibration to be performed more cost-effectively, quickly, and reliably. Operations in the vicinity of the infrastructure multi-sensor facility can be maintained with less disruption during sensor calibration. Sensor calibration can be automated and performed regularly. The insertion of markers specifically used for calibration can be eliminated.

[0008] The infrastructure facility can be stationary. The infrastructure facility can be used for safety monitoring and / or traffic monitoring, preferably in road traffic. The infrastructure facility can include a roadway.

[0009] The infrastructure multiple sensor device can be fixed in a stationary position. The infrastructure multiple sensor device can detect at least one vehicle, preferably a motor vehicle, in particular on the roadway. The infrastructure multiple sensor device can be connected to at least one vehicle for communication. The vehicle can be equipped with a driver assistance system. The vehicle can be operated semi-autonomously or autonomously.

[0010] The environment object can be a stationary or moving object. The first environment object can be a road marking, a building, a vehicle, a street sign, or a bridge.

[0011] The first sensor can be a camera, a lidar, a stereo camera, or a radar. The second sensor can be a camera, a lidar, a stereo camera, or a radar. The first and second sensors can use the same or different measurement principles. For example, the first sensor can be a camera and the second sensor can be a lidar or a camera.

[0012] The first and second sensors may be attached to a common bracket of the infrastructure multi-sensor device.

[0013] Sensor calibration can enable a spatial classification of the first sensor in the infrastructure multi-sensor setup. This allows the first sensor to be positioned in a geometric reference system. This allows a sensor position or pose to be determined and a coordinate transformation to be performed between the sensor reference system and a global coordinate system, for example, into Cartesian (x, y, z) or world coordinates (latitude, longitude, altitude) of the infrastructure setup.

[0014] The first object parameter may be a distance, an orientation, a pose, an extent, a speed, an acceleration, a rotational acceleration, a rotational speed, a tilt, or a temperature of the first environment object.

[0015] In a preferred embodiment of the invention, it is advantageous if the sensor calibration is an initial calibration upon initial commissioning of the first sensor in the infrastructure multi-sensor device or an operational calibration during operation of the first sensor in the infrastructure multi-sensor device as part of a monitoring measure. During the initial calibration, the first sensor can be adapted to the second sensor in the infrastructure multi-sensor device. This allows the first sensor of the infrastructure multi-sensor device to be adapted more easily and quickly.

[0016] During operational calibration, potential deviations in the first sensor can be detected and compensated for. The monitoring measure can perform operational calibration at specified intervals. Operational calibration can be performed online, i.e., with the first sensor inserted and connected in the infrastructure multi-sensor device. The monitoring measure can be performed in real time. The infrastructure facility can operate more smoothly. For example, interruptions in the operation of the infrastructure facility can be avoided. In particular, road closures can be prevented.

[0017] In a specific embodiment of the invention, it is advantageous if the assignment comparison is performed using a comparison algorithm that utilizes machine learning. The comparison algorithm can be trained using deep learning.

[0018] A preferred embodiment of the invention is advantageous in that, during sensor calibration, at least one additional second object parameter of the first environmental object is detected, placed in the respective sensor reference system, and taken into account in the assignment comparison. This allows the calibration to be performed more accurately. The first object parameter can be a distance to the respective sensor, and the second object parameter can be a dimension of the first environmental object in at least one first direction, or vice versa.

[0019] In a specific embodiment of the invention, it is advantageous if the first object parameter is detected by at least one third sensor, referenced to its third sensor reference system in a further assignment, and compared with the first and / or second assignment via a further assignment comparison if the further assignment comparison detects a deviation between the first and second assignments. This can increase the accuracy of the calibration. The third sensor may already have undergone a confirmed correct calibration.

[0020] In a preferred embodiment of the invention, it is advantageous if the second sensor has already been calibrated by an upstream sensor calibration prior to the sensor calibration. The second sensor can enable confirmed, reliable detection through the upstream sensor calibration. In a preferred embodiment of the invention, it is provided that the first and second sensors are each based on a different measuring principle. The first environmental object selected for sensor calibration can be a stationary object or a moving object. If the first sensor is a camera and the second sensor is a lidar, or vice versa, then the first environmental object is preferably a stationary object, in particular a building, a roadway, or a road marking.If the first sensor is a camera and the second sensor is a radar, or vice versa, then the first environmental object is preferably a moving object, in particular a vehicle, and the first object parameter is a bounding box or a detection point of the first environmental object. If the first sensor is a radar and the second sensor is a lidar, or vice versa, then the first environmental object is preferably a moving object, in particular a vehicle, and the first object parameter is a bounding box or a detection point of the first environmental object.

[0021] A preferred embodiment of the invention is advantageous in which the first and second sensor reference systems are each implemented as sensor coordinate systems. The sensor coordinate system can be converted into a global coordinate system of the infrastructure facility through a coordinate transformation.

[0022] Furthermore, a method for detecting an environmental object in an infrastructure facility using an infrastructure multi-sensor device is proposed, comprising at least one first sensor for detecting at least one first environmental object in the infrastructure facility. The first sensor has been calibrated by sensor calibration using a method with at least one of the aforementioned features, and the first environmental object is detected by relating a first object parameter of the first environmental object to the first sensor reference system as a first assignment. This allows the detection to be carried out more accurately.

[0023] Furthermore, an infrastructure multi-sensor device for detecting environmental data of an infrastructure facility is proposed, comprising at least one first sensor and a second sensor, each for detecting at least one first environmental object in the infrastructure facility. The first sensor is calibrated by a sensor calibration according to a method having at least one of the previously described features. This allows the infrastructure multi-sensor device to be implemented more cost-effectively.

[0024] Further advantages and advantageous embodiments of the invention will become apparent from the description of the figures and the illustrations.

[0025] The invention is described in detail below with reference to the figures. They show in detail:

[0026] Figure 1: An infrastructure facility with an infrastructure multiple sensor device in a special embodiment of the invention.

[0027] Figure 2: An infrastructure facility when carrying out a method in a specific embodiment of the invention.

[0028] Figure 3: A partial sequence of a calibration method in another specific embodiment of the invention.

[0029] Figure 4: A relationship between object parameters when using different sensors.

[0030] Figure 5: A method for calibration in another specific embodiment of the invention.

[0031] Figure 6: A method for calibration in another specific embodiment of the invention.

[0032] Figure 1 shows an infrastructure facility with an infrastructure multi-sensor device in a specific embodiment of the invention. The infrastructure facility 10 is stationary and installed, for example, on a roadway bridge 12. The infrastructure facility 10 is preferably used for safety monitoring and traffic monitoring.

[0033] The infrastructure facility 10 comprises an infrastructure multiple sensor device 14 mounted on the roadway bridge 12. The infrastructure multiple sensor device 14 is stationary and can detect an environment 16 of the infrastructure facility 10.

[0034] The environment 16 includes a roadway 18 and adjacent environmental objects 20. Furthermore, the infrastructure multiple sensor device 14 can detect vehicles 22 on the roadway 18 as environmental objects 20 of the infrastructure facility 10. The infrastructure multiple sensor device 14 comprises a first sensor 24 and a second sensor 26, each for detecting at least one first environmental object 28 in the infrastructure facility 10. The first and second sensors 24, 26 are attached to a common mount 30 of the infrastructure multiple sensor device 14. The first and second sensors 24, 26 can use the same measuring principle or a different measuring principle. The first sensor 24 can be a radar 32 and the second sensor 26 can be a camera 34.

[0035] The first sensor 24 can be used by a method 35 to detect the first environmental object 28. The first sensor 24 can detect the first environmental object 28 by relating a first object parameter 38 of the first environmental object 28 to a first sensor reference system 40 as a first assignment 42. This allows, for example, a spatial position of the first environmental object 28 to be detected.

[0036] The first environmental object 28 can be a roadway lighting 44 adjacent to the roadway 18. The first and second sensors 24, 26 can detect further environmental objects 20, in particular vehicles 22 or the road markings 46.

[0037] During operation of the infrastructure multiple sensor device 14 or during initial commissioning of the first sensor 24, calibration of the first sensor 24 may be necessary. The first sensor 24 is calibrated by detecting at least the first object parameter 38, for example a distance of a support 48 of the first environmental object 28 embodied as roadway lighting 44, from the first sensor 24 and relating it to the first sensor reference system 40 as a first assignment 42, and by detecting the first object parameter 38 from the second sensor 26 and relating it to a second sensor reference system 50 as a second assignment 52, and subsequently comparing the first and second assignments 42, 52, and calibrating the first sensor 24 based on this assignment comparison 54.

[0038] This allows the first sensor 24 to be calibrated more cost-effectively, quickly, and reliably, while maintaining the uninterrupted operation of the infrastructure facility 10. A marker specifically constructed for calibration in the environment 16 of the infrastructure facility 10 can be omitted.

[0039] For example, the first sensor 24 is calibrated during initial commissioning by reducing the difference between the first and second assignments 42, 52 as much as possible during the assignment comparison 54. Figure 2 shows an infrastructure facility during implementation of a method in a specific embodiment of the invention. The structure of the infrastructure facility 10 is similar to that in Figure 1. Here, a method 56 for calibrating the first sensor 24 is used, which can be triggered during a monitoring measure. In this case, a sensor calibration 58 can be carried out as an operational calibration 60 of the first sensor 24. If it is determined during the assignment comparison 54 that the first assignment 42 deviates from the second assignment 52, which is assumed to be correct, the first sensor 24 is calibrated in order to align the first assignment 42 as closely as possible to the second assignment 52.

[0040] Figure 3 shows a partial sequence of a calibration method in a further specific embodiment of the invention. The method 56 for calibrating the first sensor 24 and further sensors 66 can be based on the correctness of a calibration of the second sensor 26 as the starting sensor 62. The correctness of the calibration of the second sensor 26 can have been confirmed by a preceding sensor calibration 64. The calibration of the first sensor 24 and the further sensors 66 can compare the first assignment and further assignments with the second assignment and calibrate the first sensor 24 and the further sensors 66 depending on the assignment comparison. In this case, a third sensor 68 can in turn be calibrated based on the calibrated first assignment of the first sensor 24.

[0041] Figure 4 shows a relationship between object parameters when using different sensors. With a first sensor embodied as a camera 34, the surrounding objects can be detected optically. This means that the surrounding objects must be visible. The first surrounding object 28 can be, for example, a road marking. With a second sensor embodied as a lidar 69 or a stereo camera 70, requirements are placed on the geometric shape or the material property of the first surrounding object 28 in order to be able to detect it. To calibrate the sensor of the first sensor via the assignment comparison with the second sensor, a first object parameter 38 is selected that can be detected by the first and second sensors, for example a spatial parameter such as the distance of a stationary surrounding object.Even in the case of a reversed arrangement, i.e. the first sensor is designed as a lidar 69 and the second sensor as a camera 34, these requirements apply to the first environmental object 28 and the first object parameter 38.

[0042] If the first sensor is configured as a camera 34 and the second sensor as a radar 32, the first environmental object 28 should be moving and have a speed. The first object parameter 38 can be a speed, a bounding box, or a detection point of the first environmental object 28 in order to detect it. Even with a reversed arrangement, i.e., the first sensor is configured as a radar 32 and the second sensor as a camera 34, these requirements apply to the first environmental object 28 and the first object parameter 38.

[0043] If the first sensor is configured as radar 32 and the second sensor is configured as lidar 69 or a stereo camera 70, the first environmental object 28 should be moving and have a speed. The first object parameter 38 can be a speed or a trajectory of the first environmental object 28 in order to be able to detect it. Even with a reversed arrangement, i.e., the first sensor is configured as lidar 69 and the second sensor as radar 72, these requirements apply to the first environmental object 28 and the first object parameter 38.

[0044] Figure 5 shows a calibration method in a further specific embodiment of the invention. The calibration method 56 comprises, on the one hand, a measuring process 74 of the output sensor 62, here, for example, the second sensor 26, and a measuring process 76 of the first sensor 24. First, in a data acquisition step 78, sensor data from the second sensor 26, which describe the first object parameter 38, are acquired, and in a further data acquisition step 80, sensor data from the first sensor 24, which describe the first object parameter 38, are acquired. Subsequently, in an assignment step 82, the first object parameter 38 is related to a second sensor reference system of the second sensor 26, and in a further assignment step 84, the first object parameter 38 is related to a first sensor reference system of the first sensor 24. The second assignment is compared with the first assignment via the assignment comparison 54. During the assignment comparison 54, it is also checked whether the first

[0045] Object parameter 38 is a matching first object parameter 38 from both sensors 24, 26.

[0046] In a subsequent correction step 86, the first assignment is adjusted to the second assignment. Depending on the assignment comparison 54, the difference between the first and second assignments is compared in a verification step 88. If necessary, the process in the assignment comparison 54 is repeated, or the first sensor 24 is output as calibrated in a release step 90.

[0047] Figure 6 shows a calibration method in a further specific embodiment of the invention. The calibration method 56 as an operational calibration 60 can be carried out following an upstream sensor calibration 91 as an initial calibration 92 during initial commissioning of the first sensor. A control operation 94 of the first sensor for detecting environmental objects following the operational calibration 60 of the first sensor can regularly include a monitoring measure 96 in which the accuracy of the first sensor is checked and, in the event of excessive

[0048] Deterioration is improved by the operational calibration 60 or, if the requirements are met, normal operation 94 is continued.

Claims

Patent claims 1. Method (56) for calibrating an infrastructure multiple sensor device (14) for acquiring environmental data of an infrastructure facility (10), comprising at least one first sensor (24) and a second sensor (26), each for acquiring at least one first environmental object (28) in the infrastructure facility (10), wherein the first sensor (24) is calibrated by a sensor calibration (58) in which at least one first object parameter (38) of the first environmental object (28) is acquired by the first sensor (24) and related to a first sensor reference system (40) as a first assignment (42), and the first object parameter (38) is acquired by the second sensor (26) and related to a second sensor reference system (50) as a second assignment (52), and subsequently the first and second assignments (42, 52) are compared and the first sensor (24) is calibrated depending on this assignment comparison (54).

2. Method (56) for calibration according to claim 1, characterized in that the sensor calibration (58) is an initial calibration (92) during the initial commissioning of the first sensor (24) in the infrastructure multiple sensor device (14) or an operational calibration (60) during operation of the first sensor (24) in the infrastructure multiple sensor device (14) as part of a monitoring measure (96).

3. Method (56) for calibration according to claim 1 or 2, characterized in that the assignment comparison (54) is carried out by applying a comparison algorithm in which machine learning is used.

4. Method (56) for calibration according to one of the preceding claims, characterized in that during sensor calibration at least one further second object parameter of the first environmental object (28) is detected, classified in the respective sensor reference system (40, 50) and taken into account in the assignment comparison (54).

5. Method (56) for calibration according to one of the preceding claims, characterized in that the first object parameter (38) is detected by at least one third sensor (68), is further related to its third sensor reference system and is compared with the first and / or second via a further comparison of the assignments. The assignment (42, 52) is compared if the further assignment comparison detects a deviation between the first and second assignment (42, 52). Method (56) for calibration according to one of the preceding claims, characterized in that the second sensor (26) has already been calibrated by a prior sensor calibration (91) before the sensor calibration (58). Method (56) for calibration according to one of the preceding claims, characterized in that the first and second sensor (24, 26) are each based on a different measuring principle. Method (56) for calibration according to one of the preceding claims, characterized in that the first and second sensor reference systems (40, 50) are each implemented as sensor coordinate systems.Method (35) for detecting an environmental object (28) in an infrastructure facility (10) with an infrastructure multiple sensor device (14), comprising at least one first sensor (24) for detecting at least one first environmental object (28) in the infrastructure facility (10), wherein the first sensor (24) was calibrated by a sensor calibration (58) using a method according to one of the preceding claims and the first environmental object (28) is detected by relating a first object parameter (38) of the first environmental object (28) to the first sensor reference system (40) as a first assignment (42).Infrastructure multi-sensor device (14) for acquiring environmental data of an infrastructure facility (10) and comprising at least one first sensor (24) and one second sensor (26) each for acquiring at least one first environmental object (28) in the infrastructure facility (10), wherein the first sensor (24) is calibrated by a sensor calibration (58) according to a method (56) of claims 1 to 8.

Images