Method and system for automatic calibration of a sensor

Abstract

The present invention relates to a method for automatically calibrating sensors for a vehicle, wherein at least one first passive optical sensor and at least one second active optical sensor are calibrated by a calibration unit based on a matching spatial orientation of identified environmental features in transformed sensor data of the first sensor and sensor data captured by the second sensor.

Description

Field of the Invention

[0001] The present invention relates to a method for automatic calibration of sensors, a system for performing such a method, and a vehicle equipped with the system. Background Art

[0002] In recent years, in various application fields, the number of sensors for capturing various measured variables has increased significantly. In this way, the physical and chemical properties of the environment can be qualitatively and quantitatively captured using measured variables in the form of sensor data. The combined arrangement of several sensors (especially different sensors) also plays an increasingly important role. This applies in particular to automated processes and their systems, in which various measured variables must be related to each other.

[0003] One such example is a general vehicle, especially those designed for autonomous driving. Today, there are a large number of different sensors in modern cars, which on the one hand assist the driver and on the other hand ensure the safety of the driver and other road users. These sensors are usually combined in a so-called driver assistance system ("Advanced Driver Assistance System"; abbreviated as ADAS). The correct matching (i.e., calibration) of the sensors with each other is crucial for ensuring the function of the above-mentioned system.

[0004] It is known from the prior art that optical sensors (such as cameras), which represent most of the sensors arranged on a vehicle, are jointly calibrated in the factory. The vehicle is arranged in a stationary position in a specially designed calibration table, and each individual camera is calibrated to compensate for lens errors and to determine the position and orientation of the camera coordinate system (local coordinate system) in a higher-level world coordinate system. In addition, the relative orientation of the cameras with respect to each other is calibrated by relative spatial translation and rotation of the camera coordinate systems or camera images, and known methods are used for this purpose. For calibration, a stationary and easily recognizable pattern around the vehicle, such as a chessboard pattern, is usually used.

[0005] The above calibration method has several disadvantages. Calibration requires a specially designed calibration table, which must be set up correctly and must provide sufficient space. The calibration must be performed by technicians who have been specially trained for this purpose, which is laborious and can be said to be performed manually, where the calibration specifically includes a large number of steps that must be continuously performed manually in order to obtain sufficient calibration quality. In addition, whenever the system changes, such as when a component is replaced, the sensors must be calibrated again. Such calibration is complex, costly, time-consuming, and difficult to implement in the workshop after factory delivery, because appropriately trained and educated personnel and an appropriate calibration table are required to perform the necessary repeated calibration.

[0006] Therefore, an object of the present invention is to provide a method and a system that overcome the above disadvantages of the prior art Summary of the Invention

[0007] This object is achieved by a method for automatic calibration of sensors for a vehicle, the method comprising the following steps:

[0008] a. Capturing sensor data related to the vehicle environment through which the vehicle passes during operation by at least one first passive optical sensor arranged on the vehicle and at least one second active optical sensor arranged on the vehicle;

[0009] b. Calibrating at least one first sensor by a calibration unit determining internal sensor parameters and distortion parameters based on the sensor data captured by the first sensor, and applying the internal sensor parameters and distortion parameters to the sensor data captured by the first sensor to obtain transformed sensor data;

[0010] c. Identifying environmental features of the previously passed vehicle environment in the transformed sensor data of the first sensor and the sensor data captured by the second sensor by an identification unit; and

[0011] d. Calibrating at least one first sensor and at least one second sensor by the calibration unit based on the matching spatial directions of the identified environmental features in the transformed sensor data of the first sensor and the sensor data captured by the second sensor, while determining external sensor parameters and applying the external sensor parameters to the transformed sensor data from the first sensor and the sensor data captured by the second sensor to obtain calibrated sensor data respectively.

[0012] The vehicle in the sense of the present invention may specifically be any motor vehicle, which is preferably driven by an internal combustion engine, an electric motor, and / or a fuel cell. In addition, the vehicle is also understood to specifically refer to those vehicles that are provided and designed for autonomous driving.

[0013] According to the present invention, the method is performed while the vehicle is in operation. Operation is understood to mean that the vehicle is in motion or traveling, where the engine of the vehicle is active and propels the vehicle. The vehicle is in a normal operating mode, that is, the vehicle is being driven by a driver. Therefore, during operation, the position of the vehicle changes continuously according to traffic conditions, road conditions, etc.

[0014] The vehicle environment through which the vehicle passes during operation is understood to refer to the environment located around the vehicle, through which the vehicle drives or moves. This may preferably relate to any object, such as buildings, vegetation, infrastructure, etc. It is preferably assumed that the sensors according to the present invention are provided and designed to be able to capture the vehicle environment and output sensor data related to the vehicle environment.

[0015] It should still be noted that, according to the present invention, it is assumed that when the method is executed, the vehicle is in motion or being driven. The method according to the present invention is clearly not executed in a stationary calibration table specifically provided for this purpose. In addition, the vehicle environment should not be an intentionally arranged environment in a calibration table specifically provided for sensor calibration. Furthermore, automatic calibration is understood to mean that the calibration is automatic during vehicle operation, that is, executed in a self-controlled manner, and does not require specially trained personnel to execute the method or individual steps. The sequence of the calibration method and its quality or success are preferably presented or indicated to the user or vehicle driver in a manner that can be understood by the user or vehicle driver without having special knowledge. Automatic calibration also means that the calibration can preferably be started, stopped, restarted, etc. by the user.

[0016] The first sensor in the sense of the present invention is a passive sensor, and the second sensor is an active sensor. Existing sensors provide sensor data. The sensors on the vehicle or of the vehicle are preferably sensors arranged in or on the vehicle. Such sensors can be part of the vehicle, or can be integrated in and / or on the vehicle, for example as part of a driver assistance system, or can be subsequently installed in and / or on the vehicle. It is assumed that the first sensor and the second sensor are different sensors, that is, the first sensor and the second sensor capture different measurement variables.

[0017] Preferably, the sensor data is continuously captured during vehicle operation according to step a, that is, new sensor data is always captured, and / or at specific time intervals and / or when a specific event occurs. For example, the event can be activated by a vehicle passenger.

[0018] According to the present invention, at least one first sensor is calibrated so as to first map the captured sensor data to the real world (preferably, transform two-dimensional sensor data into a three-dimensional captured environment), and also compensate for any errors (such as distortion, etc.) that occur when the sensor data is captured by the sensor and represent the difference between the vehicle environment in the sensor data and the real vehicle environment. As a result of the calibration, the sensor data of at least one first sensor can be used in a further method in the form of transformed sensor data. The calibration is performed by determining internal sensor parameters and distortion parameters based on the sensor data captured by the first sensor, preferably based on a mathematical model or by an algorithm. The internal sensor parameters particularly indicate the position of the sensor relative to the sensor data (related to the image measurement variables of an optical sensor) and the position and orientation of the sensor coordinate system in a higher-level world coordinate system (vehicle environment), or the position of the sensor relative to the captured sensor data (in other words, the recorded object in the sensor data).

[0019] The distortion parameters are determined based on a known sensor model (such as a lens model) of a first sensor corresponding to captured sensor data, and are used to correct imaging errors (distortion) caused by the structure of the sensor itself (including the lens equation) and environmental influences (such as temperature, weather, etc.). In addition, the distortion parameters are used to correct errors caused by mechanical influences. For example, the mechanical influences can be caused by vibrations caused by the movement of the vehicle during operation and also affect the sensor. In the context of the present invention, according to the present invention, when the vehicle is in operation, i.e., the vehicle is moving, automatic calibration of the sensor is performed, and the correction of the mechanical influences caused by the distortion parameters is particularly relevant, and in this way makes the method more robust under given conditions and more capable of being successfully performed than known methods.

[0020] By applying the internal sensor parameters and the distortion parameters to the sensor data captured by the first sensor, transformed sensor data are obtained, and these transformed sensor data are transformed or corrected such that they truly reproduce the vehicle environment and are suitable for further use in the method. It can be said that the transformed sensor data are two-dimensionally captured sensor data that are converted or transformed back to three-dimensional reality (or world coordinate system). For the calibration of the first sensor, it preferably includes the spatial orientation and movement of the vehicle, which can be estimated (preferably by a known odometer) and / or captured as measurement variables (preferably GNSS data or also GPS data with RTK (real-time kinematic carrier-phase differential)).

[0021] The calibration of the first sensor is preferably performed based on sensor data captured from different perspectives. Here, it is particularly advantageous that the vehicle passes through the vehicle environment, i.e., drives through the vehicle environment, and thus sensor data that are temporally continuous are automatically captured from different perspectives. The calibration of the first sensor is particularly preferably based on a mathematical (camera) model or algorithm, such as an (extended) pinhole camera model, a fish-eye model, resection, and / or bundle adjustment, where the model or algorithm is not limited to these examples. Preferably, specifically during the calibration process by bundle adjustment, environmental features of the previously passed vehicle environment can be identified, where sensor data from different perspectives are advantageously included, and where, for this purpose, the spatial orientation and movement of the vehicle are estimated and / or measurement data are used. A method similar to step d is preferably used for the identification of environmental features as shown below, or another method can also be used.

[0022] As an example, reference is made here to the calibration of a camera, which is performed in a known manner by determining the internal and external orientations (internal sensor parameters). These camera calibrations and their working methods are well known in the prior art and should be considered to be disclosed in the context of the present invention.

[0023] The environmental features are understood to be the objects in the vehicle environment captured by the sensors. For example, the environmental features are buildings, vegetation, etc. arranged in the vehicle environment.

[0024] Based on the matching spatial orientation of the sensor data transformed by the first sensor and the identified environmental features in the sensor data captured by the second sensor, and the determination of the external sensor parameters, the calibration of at least one first sensor and at least one second sensor by the calibration unit is understood as the relative spatial translation and / or rotation of the sensor data transformed by the first sensor with respect to the sensor data of the second sensor or the corresponding known sensor coordinate system. The external sensor parameters include a specific rotation matrix and / or a specific translation matrix reflecting the rotation and / or translation of the sensor data. The rotation and / or translation operations are performed in such a way that the environmental features identified in the sensor data transformed by the first sensor and the sensor data of the second sensor are spatially arranged or oriented in the same way (also referred to as "homogeneous transformation"). Thus, the difference in the spatial orientation of the identified environmental features is minimized, which is preferably achieved by the calibration unit minimizing the corresponding mathematical function. In this way, the relative orientation or alignment of at least one first sensor and at least one second sensor can be determined. The calibrated sensor data is obtained by applying the external sensor parameters to the sensor data of the first sensor and the second sensor, wherein the calibrated sensor data is related to the coordinate system common to at least one first sensor and at least one second sensor, and wherein the objects, articles, and / or persons are in the same position in the sensor data of all sensors. The calibration unit is provided and designed to perform the calibration of the first sensor and the second sensor according to the present invention based on the minimization of a mathematical function that specifies the deviation of the spatial orientation of the environmental features. Then, the sensor data of the sensors calibrated according to the present invention is used for all further evaluations or by all further systems on the vehicle.

[0025] The method according to the present invention ensures that different optical sensors arranged at different positions in and / or on the vehicle can be calibrated to a common sensor coordinate system by spatially matching the orientation of the environmental features in the captured sensor data, and the captured sensor data relates to the passing vehicle environment. Since the sensors are arranged in and / or on the vehicle in a stationary manner, the difference in the spatial orientation of the identified environmental features is only caused by the different positioning of the sensors in and / or on the vehicle. Advantageously, the environmental features in the vehicle environment through which the vehicle passes or drives are used for calibration, wherein a special calibration table or the like can be dispensed with, and the calibration can be performed during the normal operation of the vehicle. This calibration is particularly relevant to the sensors of the driver assistance system if pedestrians and / or cyclists are to be clearly identified.

[0026] According to a preferred embodiment, in step a, the sensor data is captured simultaneously by the first sensor and the second sensor. The calibration of the first sensor and the second sensor in step d is preferably performed based on the simultaneously captured sensor data. In this way, the different spatial orientations of the environmental features in the sensor data of the first sensor and the second sensor are only caused by the different positions of the sensors in and / or on the vehicle.

[0027] According to a preferred embodiment, the environmental feature is at least a part of a substantially static object, which substantially static objects are arranged in the vehicle environment through which the vehicle passes during operation, and are detected by at least one first sensor and at least one second sensor using sensor data. The stationary objects in the vehicle environment are preferably objects arranged in a stationary manner, that is, their spatial orientation or position in the vehicle environment is substantially unchanged. Therefore, it can be preferably assumed that the different spatial orientations of the environmental features are only caused by the different positions of the sensors. Such static objects in the vehicle environment are preferably buildings, vegetation, etc. The static objects themselves are not recognized, but parts or portions of the static objects are recognized, for example, by clearly defined corners and / or edges of the building or high-contrast color differences. It should be clear that the static objects are not objects specifically for calibration, such as checkerboard patterns, circular markers or barcodes, which are set in the vehicle environment specifically for the purpose of sensor calibration. Preferably, the recognition unit recognizes the environmental features by a gradient-based method or a key-point-based method. In the case of gradient-based recognition ("gradient direction measurement"), color, contrast and / or intensity gradients are preferably used to recognize the environmental features in the sensor data. Preferably, the environmental features are recognized in key-point-based recognition based on the SIFT method ("scale-invariant feature transform"). The functions and applications of the gradient-based method and the key-point-based method are well known in the prior art. It is also conceivable to use other suitable methods to replace or supplement the gradient-based method or the key-point-based method.

[0028] According to a preferred embodiment, at least one first sensor is designed as a monocular camera or a stereo camera. The at least one second sensor preferably uses radar, lidar or ultrasonic technology, or is designed as a time-of-flight (TOF) camera. A plurality of first sensors and second sensors can be provided, wherein each first sensor is calibrated with each second sensor according to the method according to the present invention. Therefore, the first sensor preferably provides sensor data in the form of a 2D image, and the second sensor provides sensor data in the form of a point cloud or a point group or a depth map. These types of sensors are specifically used for driver assistance systems and are also very important for autonomous driving, which is why their calibration is very important. It is conceivable that other sensor types not explicitly mentioned here can also be used for this method.

[0029] Preferably, any number of first sensors and second sensors are provided, wherein each first sensor is calibrated with each second sensor according to the method shown.

[0030] According to a preferred embodiment, steps a to c are repeatedly performed continuously. In step c, the identified environmental features, if identified as matching in a plurality of sensor data captured continuously in terms of time and / or position, are preferably stored as consistent environmental features by the identification unit or deleted. Preferably, step d is only performed when the number of consistent environmental features exceeds a predetermined first threshold. The number of consistent environmental features is preferably compared with the threshold by the evaluation unit to determine whether the value is exceeded. In this way, the calibration of the sensors is only performed based on the consistent environmental features identified as matching in a plurality of continuously captured sensor data. The number of continuously captured sensor data constituting the consistent environmental features can preferably be defined as needed. The environmental features evaluated as inconsistent are deleted as untrustworthy and not used for further calibration. The sensor data captured continuously in terms of time is preferably understood, for example, as the sensor data captured continuously when driving through the vehicle environment. The sensor data captured continuously in terms of position is preferably understood as the sensor data captured at the same position or from the same vehicle position, wherein it is assumed here that in the case of sensor data captured continuously in terms of position, the sensors pass through the same vehicle environment and thus can identify substantially the same environmental features. The matching identification is preferably understood to mean that the environmental features in the sensor data match in terms of space or position, wherein it can be assumed that the environmental features are at least part of a substantially static object, the spatial orientation of which does not change relative to the sensor data. This enables the sensor data to be advantageously calibrated because the differences in the spatial orientation of the environmental features are basically only caused by different sensor positions. The predetermined first threshold is an arbitrarily determinable value or number of consistent environmental features, wherein the higher the threshold, the more reliable the calibration, but many consistent environmental features must be identified.

[0031] According to a preferred embodiment, the distribution parameter is determined by an evaluation unit, which indicates the spatial distribution of consistent environmental features in the respective sensor data. Preferably, step d is only executed if the number of consistent environmental features exceeds a first threshold and if the value of the distribution parameter exceeds a predetermined second threshold. The distribution parameter preferably reflects how the consistent environmental features are spatially distributed over / within the sensor data, where the larger the spatial distribution, the larger the distribution parameter. It is also conceivable to specify the distribution parameter as a coverage parameter, in which case the distribution of the environmental features over / within the sensor data can also be referred to as the coverage of the sensor data with environmental features. Assuming that due to the larger distribution of consistent environmental features over / within the sensor data, which involves multiple static objects, a more reliable calibration is possible, as opposed to the case where the identified environmental features are restricted to a spatially limited area. This embodiment ensures that step d is only executed when a certain number of consistently distributed environmental features are identified.

[0032] According to a preferred embodiment, a third sensor is provided on the vehicle. The third sensor is preferably provided and designed to capture the spatial orientation and movement of the vehicle. In step b, the captured spatial orientation and movement of the vehicle are also preferably used to calibrate at least one first sensor. More preferably, the third sensor is a GNSS receiver. In the case of calibrating the first sensor according to step b, the spatial orientation and movement of the vehicle are sometimes required, where this can be estimated using sensor data (from different perspectives). However, sensor data that accurately reflects the spatial orientation and movement of the vehicle is more suitable for this purpose and thus contributes to a more accurate calibration.

[0033] According to a preferred embodiment, the calibration unit, the identification unit, and / or the evaluation unit are arranged on or outside the vehicle. The calibration unit, the identification unit, and / or the evaluation unit can particularly preferably be designed as part of a common computing unit. The calibration unit, the identification unit, and / or the evaluation unit are preferably designed outside the vehicle, as part of a data processing system, or based on the cloud. The internal sensor parameters, the external sensor parameters, the distortion parameters, the consistent environmental features, and / or the distribution parameters can preferably be stored in a retrievable manner in a storage unit arranged on or outside the vehicle. The calibration unit, the identification unit, and / or the evaluation unit on the vehicle are preferably designed as part of an existing driver assistance system or subsequently arranged in and / or on the vehicle.

[0034] The data processing system in the sense of the present invention should be understood as an IT infrastructure, which specifically includes memory, computing power, and application software (if applicable). The data processing system according to the present invention is preferably established and provided for receiving, sending, processing, and / or storing data. Accordingly, an external data processing system is a data processing system arranged outside the vehicle.

[0035] According to a preferred embodiment, the calibration unit, the recognition unit, and / or the evaluation unit, or the data processing system and / or the cloud are designed to use artificial intelligence (AI) and preferably neural networks to calibrate the sensor (steps b and d) and to recognize environmental features (step c). The terms "machine learning" and "deep learning" related to the application of artificial intelligence (AI) and neural networks should be advantageously mentioned here.

[0036] According to a preferred embodiment, based on the number of consistent environmental features relative to a first threshold and the value of the distribution parameter relative to a second threshold, the sequence of the method is displayed on a display device. In this way, the user (possibly a vehicle passenger) can follow the sequence of the method. More preferably, an operating device is provided by which the user can adjust the sequence of the method. The display device can be a screen permanently installed in the vehicle (such as a multimedia system and / or a vehicle navigation system), a smartphone, a tablet computer, and / or a laptop computer, but this list should not be construed as exhaustive. In the case of an autonomous or remotely controlled vehicle, the display device is particularly preferably arranged outside the vehicle. The display device allows the user (such as the driver and / or a vehicle passenger) to follow the calibration process, in which, based on the number of recognized or stored consistent environmental features and the value of the distribution parameter, the sequence or progress is displayed relative to the corresponding threshold. In addition, the method can be adjusted by an operating device (such as as part of a touch screen), where the method can be started, stopped, interrupted, or restarted.

[0037] According to a preferred embodiment, the sensor data and / or various parameters are sent from the vehicle to the data processing system or the cloud via a wireless connection and vice versa, the wireless connection preferably being based on a transmission technology selected from the group including WLAN connection, radio connection, mobile radio connection, 2G connection, 3G connection, GPRS connection, 4G connection, 5G connection. The wireless connection advantageously has a relatively long range, at least in part, preferably having a maximum range of more than 100 m, preferably more than 500 m, preferably more than 1 km, and particularly preferably several km. In this way, data can be sent from the vehicle to the data processing system / cloud and vice versa regardless of the corresponding geographical location. The wireless connection or transmission technology is preferably a two-way connection.

[0038] This object is also achieved by a system for performing the method according to any one of the preceding claims, the system comprising at least one first sensor and at least one second sensor, a calibration unit, and a recognition unit.

[0039] At least in terms of signaling, the sensor is preferably connected to a calibration unit and an identification unit. More preferably, at least in terms of signaling, the sensor is also connected to an evaluation unit. The calibration unit, the identification unit, and / or the evaluation unit are also preferably connected at least in terms of signaling. The calibration unit, the identification unit, and / or the evaluation unit are particularly preferably designed as part of a computing unit. The calibration unit, the identification unit, and the evaluation unit or the computing unit are preferably connected at least in terms of signals to a data processing system external to the vehicle or the cloud.

[0040] The sensor is preferably provided and designed to capture the vehicle environment based on the measured variable, output sensor data related to the vehicle environment, and send it to the appropriate unit.

[0041] The calibration unit is preferably provided and designed to calibrate at least one first sensor by determining internal sensor parameters and distortion parameters based on the sensor data captured by the first sensor, where a mathematical model or algorithm is preferably used for this purpose. In addition, the calibration unit is preferably provided and designed to apply the determined internal sensor parameters and distortion parameters to the sensor data of the first sensor, and thus obtain transformed sensor data that can be used for further methods. Furthermore, the calibration unit is provided and designed to calibrate at least one first sensor and at least one second sensor based on the matching spatial directions of the identified environmental features in the transformed sensor data of the first sensor and the sensor data captured by the second sensor, while determining external sensor parameters and applying the external sensor parameters to the transformed sensor data of the first sensor and the sensor data obtained by the second sensor to obtain calibrated sensor data respectively.

[0042] The identification unit is preferably provided and designed to identify environmental features of the previously passed vehicle environment in the transformed sensor data of the first sensor and the sensor data obtained by the second sensor. This identification is preferably based on a mathematical model or algorithm, where these mathematical methods are known in the prior art. The identification unit is also provided and designed to determine / identify and store consistent environmental features.

[0043] The evaluation unit is preferably provided and designed to determine a distribution parameter that specifies the spatial distribution of the consistent environmental features in the respective sensor data. In addition, the evaluation unit is provided and designed to compare the number of consistently stored environmental features with a first threshold and compare the value of the distribution parameter with a second threshold to determine whether they are exceeded, and if the value is exceeded, initiate the step or cause the step to be executed.

[0044] This object is also achieved by a vehicle equipped with a system for performing the method according to the invention.

[0045] It is conceivable that the system is subsequently arranged on and / or in the vehicle, or has been integrated into the vehicle. In addition, it is conceivable that the calibration unit, the recognition unit, and the evaluation unit or the calculation unit are designed as part of an existing calculation unit or as a separate calculation unit provided subsequently.

[0046] The features described in the method are intended to make the necessary modifications to the system and the vehicle, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Additional objects, advantages, and conveniences of the present invention can be found in the following description in conjunction with the drawings. In the drawings:

[0048] Figure 1 A vehicle in a vehicle environment for calibrating a sensor according to a preferred embodiment of the present invention is shown;

[0049] Figure 2 is a schematic diagram of a system for calibrating a sensor according to a preferred embodiment;

[0050] Figure 3 A method for calibrating a sensor using a flowchart according to a preferred embodiment is shown;

[0051] Figure 4 A display device showing the method sequence according to the present invention according to a preferred embodiment is shown. DETAILED DESCRIPTION

[0052] Figure 1 A vehicle 1 according to the present invention is shown, which is equipped with a system 1000 according to the present invention. The system 1000 includes a sensor to be calibrated arranged on the vehicle. The vehicle 1 travels or moves in a vehicle environment 4 in a moving direction R (represented by an arrow). For example, a plurality of buildings and trees are arranged in the vehicle environment.

[0053] System 1000 is designed as a so-called roof box, which is subsequently mounted on the vehicle roof. System 1000 has four first passive optical sensors 2a, 2b, 2c, 2d, which are designed as cameras. In the direction of travel, sensor 2a faces forward, sensors 2c, 2c are laterally perpendicular to the direction of travel, and sensor 2d faces rearward, opposite to the direction of travel. In this way, the entire vehicle environment 4 around the vehicle 1 can be captured by sensors 2a-d (cameras). System 1000 also has three second active optical sensors 3a, 3b, 3c, which are designed as lidar sensors (distance sensors). Here, sensor 3a is designed as a 360° lidar sensor with 32 layers and faces forward (capture area) in the direction of travel. Sensors 3b, 3c are designed as 360° lidar sensors with 16 layers and face forward (capture area) in the direction of travel. The shown sensor design and arrangement are merely examples and can also be implemented differently.

[0054] The system 1000 according to Figure 1 also includes a calibration unit 6 and an identification unit 7 (see Figure 2 ), which are not shown.

[0055] Two environmental features 5a, 5b are shown as examples in the vehicle environment 4. These two environmental features 5a, 5b can be identified by the identification unit 7 in the sensor data of the corresponding first sensors 2a-d and second sensors 3a-c as the vehicle 1 travels through the vehicle environment 4 in the direction of movement R. Environmental feature 5a is part of a building, more precisely the window facade or the transition from the window to the masonry. Environmental feature 5b relates to plants, more precisely a tree. The environmental features 5a, 5b are characterized by being different in color and / or structure and are thus easily identifiable.

[0056] Figure 2 System 1000 according to a preferred embodiment is schematically shown.

[0057] System 1000 includes at least one first sensor 2, at least one second sensor 3, and a third sensor 8, wherein the sensors 2, 3, 8 are arranged in and / or on a vehicle 1. The first sensor 2 is a passive optical sensor, such as a camera, the second sensor 3 is an active optical sensor, such as a lidar, radar, or ultrasonic sensor, and the third sensor 8 is a position sensor, such as a GNSS receiver. The sensors 2, 3, 8 are each provided and designed to capture sensor data including corresponding measured variables. For this purpose, the third sensor 8 is provided, and the third sensor 8 is designed to capture the spatial orientation and movement of the vehicle 1, wherein the sensor data of the third sensor 8 can also be used to calibrate the first sensor 2. The sensor data is sent via at least one signaling connection 14 to a computing unit 11, wherein the computing unit 11 includes a calibration unit 6, an identification unit 7, and an evaluation unit 9. The processing unit 11 can be arranged on the vehicle side and designed as part of a driver assistance system or designed independently (preferably designed as a general vehicle computing unit). In addition, the computing unit 11 can be designed as part of a data processing system or cloud arranged outside the vehicle.

[0058] The calibration unit 6 is provided and designed to calibrate at least one first sensor 2 by: determining internal sensor parameters and distortion parameters based on the sensor data captured by the first sensor 2, wherein a mathematical model or algorithm is preferably used for this purpose to compensate for errors present in the sensor data capture; and making the sensor data of the first sensor 2 available for further methods. In addition, the calibration unit 6 is provided and designed to apply the determined internal sensor parameters and distortion parameters to the sensor data of the first sensor 2, thereby obtaining transformed sensor data. In addition, the calibration unit 6 is provided and designed to calibrate at least one first sensor 2 and at least one second sensor 3 based on the matching spatial orientation of the identified environmental features 5a, 5b in the transformed sensor data of the first sensor 2, calibrate the sensor data captured by the second sensor 3 by determining external sensor parameters, and apply the external sensor parameters to the transformed sensor data of the first sensor 2 and the sensor data captured by the second sensor 3 to obtain calibrated sensor data, respectively.

[0059] The identification unit 7 is set and designed to detect environmental features 5a, 5b of a previously traversed vehicle environment 4 in the transformed sensor data of the first sensor 2 and the sensor data obtained by the second sensor 3. The identification is preferably performed based on a mathematical model or algorithm, which is preferably gradient-based or keypoint-based. The identification unit 7 is also provided and designed to determine and store consistent environmental features.

[0060] The evaluation unit 9 is preferably provided and designed to determine a distribution parameter that specifies the spatial distribution of the consistent environmental features 5 in the respective sensor data. Furthermore, the evaluation unit 9 is provided and designed to compare the values of the multiple consistent environmental features 5 and the distribution parameter with the respective assigned predetermined thresholds, and to initiate step d only when the threshold is exceeded.

[0061] The calculation unit 11 is connected to the storage unit 10 via a two-way signaling connection 15. Internal sensor parameters, external sensor parameters, distortion parameters, distribution parameters, consistent environmental features, and / or sensor data can be stored in the storage unit 10 in a retrievable manner.

[0062] Furthermore, the calculation unit 11 is connected to the display device 12 having the operating device 13 via a two-way signaling connection 15, wherein the display device is arranged in the vehicle 1. The display device 12 is provided and designed to display the sequence of the method based on the number of consistent environmental features relative to a first threshold and the value of the distribution parameter relative to a second threshold, such that the sequence of the method can be followed by a user (e.g., a vehicle passenger). The sequence of the method can be adjusted by the user via the operating device 13.

[0063] Figure 3 The preferred embodiment of the method 100 according to the present invention is illustrated using a flowchart.

[0064] The method 100 can be automatically started when the vehicle 1 is put into operation or started, or can be manually started by the user using the operating device 13.

[0065] The method starts with step S1 (corresponding to step a), and sensor data related to the vehicle environment 4 through which the vehicle passes during operation is captured by at least one first passive optical sensor 2 arranged on the vehicle 1 and at least one second active optical sensor 3 arranged on the vehicle.

[0066] Then, according to step S2 (corresponding to step b), the calibration unit 6 calibrates at least one first sensor 2 based on the sensor data captured by the first sensor 2 to determine the internal sensor parameters and the distortion parameters, and the internal sensor parameters and the distortion parameters are applied to the sensor data captured by the first sensor 2, wherein transformed sensor data is obtained.

[0067] In the subsequent step S3 (corresponding to step c), the recognition unit 6 recognizes the environmental features 5 of the previously passed vehicle environment 4 in the transformed sensor data of the first sensor 2 and the sensor data captured by the second sensor 3. Furthermore, the environmental features 5, if recognized in multiple sensor data captured continuously in terms of time and / or position, are stored as consistent environmental features 5 by the recognition unit 6 or deleted.

[0068] In a further step S4, the evaluation unit 7 determines distribution parameters which indicate the spatial distribution of the consistent environmental features 5 in the respective sensor data. This step can also be carried out in step S3. The evaluation unit 7 then compares the number of stored consistent environmental features with a predetermined first threshold value and compares the value of the distribution parameter with a predetermined second threshold value to determine whether the value is exceeded.

[0069] If in step S4 it is determined that the number of stored consistent environmental features and the value of the distribution parameter respectively exceed the assigned threshold values, step S5 is carried out. On the other hand, if in step S4 it is determined that one of the threshold values is not exceeded, the process returns to step S3 and the recognition unit 6 recognizes new or further environmental features 5 of the previously passed vehicle environment 4 in the newly transformed sensor data of the first sensor 2 and the newly captured sensor data of the second sensor 3.

[0070] According to step S5 (corresponding to step d), based on the matching spatial orientation of the recognized consistent environmental features 5 in the transformed sensor data of the first sensor 2 and the sensor data captured by the second sensor 3, the calibration unit 5 calibrates at least one first sensor 2 and at least one second sensor 3 by determining external sensor parameters. The external sensor parameters are applied to the transformed sensor data of the first sensor and the sensor data captured by the second sensor 3, thereby obtaining calibrated sensor data respectively.

[0071] In a subsequent step S6, the internal sensor parameters, the external sensor parameters, the distortion parameters, the consistent environmental features and / or the distribution parameters are stored in the storage unit 10.

[0072] When the vehicle 1 stops running or is parked, or when the user manually stops or ends the method 100 by means of the operating device 13, the method 100 automatically ends.

[0073] Figure 4 A display device 12 with an operating device 13 is shown according to a preferred embodiment.

[0074] The operating device 13 includes operating elements 16 in order to start, stop or restart the method, where further operating elements are conceivable. The operating element 16 can be designed as a touchscreen element of the display device 12 or as a mechanically operable button.

[0075] The sensor data transformed from at least one first sensor 2 in the form of a camera image 17 and the sensor data of at least one second sensor 3 in the form of a point cloud 18 are displayed on the display device 12 superimposed on each other. The identified (consistent) environmental features 5 are highlighted by markers 20 (shown here as shaded circles by way of example). In this way, the user can track the progress of the method in real time and can easily identify the identified environmental features without having to undergo specific training for this purpose.

[0076] In addition, the number of consistent environmental features relative to a first threshold and the value of the distribution parameter relative to a second threshold can each be represented as a percentage (shown as "XX%") by a calibration progress indicator 19, and an associated progress bar is arranged above this indicator. This further simplifies the identification of the sequence of the method and its progress.

[0077] All features disclosed in the application documents are considered to be essential to the invention, provided that they are novel over the prior art either individually or in combination.

[0078] List of reference signs

[0079] 1 Vehicle

[0080] 100 Method

[0081] 1000 System

[0082] 2 First sensor

[0083] 3 Second sensor

[0084] 4 Vehicle environment

[0085] 5 Environmental feature

[0086] 6 Calibration unit

[0087] 7 Identification unit

[0088] 8 Third sensor

[0089] 9 Evaluation unit

[0090] 10 Storage unit

[0091] 11 Calculation unit

[0092] 12 Display device

[0093] 13 Operating device

[0094] 14, 15 Signaling connection

[0095] 16 Operating element

[0096] 17 Camera image

[0097] 18 Point cloud

[0098] 19 Calibration progress indicator

[0099] 20 Mark

[0100] R Moving direction, traveling direction

[0101] S Step

Claims

1. A method (100) for automatic calibration of a sensor for a vehicle (1), comprising the following steps: a. Capturing sensor data related to a vehicle environment (4) through which the vehicle (1) passes during operation by at least one first sensor (2) arranged on the vehicle and at least one second sensor (3) arranged on the vehicle, wherein the first sensor is a passive optical sensor and the second sensor is an active optical sensor; b. Calibrating the at least one first sensor (2) by a calibration unit (6) by determining internal sensor parameters and distortion parameters based on the sensor data captured by the first sensor (2), and applying the internal sensor parameters and the distortion parameters to the sensor data captured by the first sensor (2) to obtain transformed sensor data; c. Identifying environmental features (5) of the previously passed vehicle environment (4) by an identification unit (7) in the transformed sensor data of the first sensor (2) and the sensor data captured by the second sensor (3); and d. Calibrating the at least one first sensor (2) and the at least one second sensor (3) by the calibration unit (6) based on a matching spatial orientation of the identified environmental features (5) in the transformed sensor data of the first sensor (2) and the sensor data captured by the second sensor (3), while determining external sensor parameters and applying the external sensor parameters to the transformed sensor data from the first sensor (2) and the sensor data captured by the second sensor (3) to obtain calibrated sensor data respectively, wherein the environmental features (5) identified in step c, if identified as matching in a plurality of sensor data captured continuously in terms of time and / or position, are stored by the identification unit (7) as consistent environmental features (5) or deleted, wherein step d is only executed when the number of the consistent environmental features (5) exceeds a predetermined first threshold, wherein a distribution parameter is determined by an evaluation unit (9), the distribution parameter representing a spatial distribution of the consistent environmental features (5) in respective sensor data, wherein step d is only executed when the number of the consistent environmental features (5) exceeds the first threshold and the value of the distribution parameter exceeds a predetermined second threshold.

2. The method (100) according to claim 1, characterized in that: the environmental features (5) are at least part of static objects arranged in the vehicle environment (4) through which the vehicle (1) passes during operation, and are detected by the at least one first sensor (2) and the at least one second sensor (3) using the sensor data, wherein the environmental features (5) are identified by the identification unit (7) by a gradient-based method or a key-point-based method.

3. The method (100) according to claim 1, characterized in that: The at least one first sensor (2) is designed as a monocular camera or a stereo camera, and the at least one second sensor (3) uses radar, lidar or ultrasonic technology or is designed as a time-of-flight (TOF) camera.

4. The method (100) according to claim 1 above, characterized in that: Steps a to c are continuously repeated.

5. The method (100) according to claim 1 above, characterized in that: A third sensor (8) is provided on the vehicle, wherein the third sensor (8) is provided and designed to capture the spatial orientation and movement of the vehicle (1), and wherein the captured spatial orientation and movement of the vehicle (1) are also used in step b for the calibration of the at least one first sensor (2), and wherein the third sensor (8) is a GNSS receiver.

6. The method (100) according to claim 1 above, characterized in that: The calibration unit (6), the recognition unit (7) and / or the evaluation unit (9) are arranged on or outside the vehicle, wherein the calibration unit (6), the recognition unit (7) and / or the evaluation unit (9) are designed as part of a data processing system or based on the cloud outside the vehicle, and wherein the internal sensor parameters, the external sensor parameters, the distortion parameters, the consistent environmental features (5) and / or the distribution parameters are stored in a retrievable manner in a storage unit (10) arranged on or outside the vehicle.

7. The method (100) according to any one of the above claims, characterized in that: Based on the number of the consistent environmental features (5) relative to the first threshold and the value of the distribution parameters relative to the second threshold, the sequence of the method (100) is displayed on a display device (12) such that a user can follow the sequence of the method (100), and wherein an operating device (13) is provided by which the user can adjust the sequence of the method (100).

8. A system (1000) for performing the method (100) according to any one of the above claims, comprising at least one first sensor (2), at least one second sensor (3), a calibration unit (6) and a recognition unit (7).

9. A vehicle (1) equipped with the system (1000) according to claim 8.

Images