Automatic external calibration and calibration verification of different sensor modalities such as camera, radar and lidar sensors

Abstract

A method for calibrating a first sensor and a second sensor of a vehicle is provided. The method comprises:-obtaining first data from the first sensor and second data from the second sensor,-filtering at least the second data based on positions of data points of the second data, and-determining one or more calibrated parameters between the first sensor and the second sensor based on the first data and the filtered second data.

Description

Technical Field

[0001] The invention relates to a method for calibrating a first sensor and a second sensor of a vehicle, and a calibration unit for calibrating a first sensor and a second sensor of a vehicle.

[0002] The invention also relates to a computer-readable storage medium having stored thereon a program code comprising instructions for executing such a method. Background Art

[0003] Modern cars, whether autonomous or not, include an increasing number of sensors of different modalities. These sensors need to be calibrated. For example, intrinsic calibration of a camera consists of determining a set of parameters of a camera projection model for associating 2D image points with 3D scene points, where the optimal parameters correspond to the smallest re-projection error. If there are sensors of more than one modality, external calibration is required. External calibration may involve determining parameters that match points from one sensor with corresponding points of another sensor. Summary of the invention

[0004] An object of the present invention is to provide a method for calibrating a first sensor and a second sensor of a vehicle and a calibration unit for calibrating the first sensor and the second sensor of a vehicle, which overcome one or more of the above-mentioned problems of the prior art.

[0005] A first aspect of the present invention provides a method for calibrating a first sensor and a second sensor of a vehicle, the method comprising:

[0006] - obtaining first data from a first sensor and second data from a second sensor;

[0007] - filtering at least the second data based on the positions of data points of the second data, and

[0008] - determining one or more parameters of a calibration between the first sensor and the second sensor based on the first data and the filtered second data.

[0009] The method of the first aspect has the advantage of performing calibration based on the filtered second data. Filtering can be performed so that obviously irrelevant points are filtered out, and calibration is performed on those points that are not filtered out. Therefore, the probability of incorrect calibration can be reduced and the overall reliability of calibration can be improved.

[0010] The first sensor and / or the second sensor may include a camera. Obtaining the first data and / or the second data may include obtaining a first image and / or a second image.

[0011] As will also be outlined below, it will be appreciated that in addition to the first and second sensors, the method may also calibrate additional sensors simultaneously.

[0012] Filtering at least the second data means that, optionally, the first data may also be filtered. Filtering at least the second data may include filtering out (i.e., removing) data points in the second data that have locations that fall within a specific one or more regions (e.g., one or more predetermined specific regions). For example, data points that have locations outside a specific target region may be removed. The target region may depend on a property of the turn, such as the turn direction of the vehicle.

[0013] The calibration may include an alignment between the first sensor and the second sensor, for example, the calibration may include three parameters for translation and a tree parameter for rotation between a coordinate system of the first sensor and a coordinate system of the second sensor.

[0014] In a first implementation of the method according to the first aspect, the method further comprises converting the first data and / or the second data into a common domain using an a priori calibration.

[0015] Using an a priori calibration has the advantage that a priori knowledge about the relative orientation between the first sensor and the second sensor can be used as a starting point for determining a final calibration between the first sensor and the second sensor. For example, knowledge about the mounting position of the first sensor relative to the mounting position of the second sensor can be used to determine the a priori calibration.

[0016] In another implementation of the method according to the first aspect, the method further comprises the steps of detecting a first object in the first data and / or detecting a second object in the filtered second data, and performing determination of one or more parameters based on the first object and / or the second object.

[0017] Performing calibration based on detected objects has the following advantage, even if the first sensor and the second sensor use different modalities (which may result in different kinds of data points in the first data and the second data), accurate calibration can be performed based on detected objects that can completely correspond in the first data and the second data.

[0018] In a further implementation of the method according to the first aspect, one or more parameters are determined based on a center point of the first object and / or a center point of the second object.

[0019] For example, a first sensor may acquire many data points for a given object, while a second sensor may require fewer data points for the same given object. However, based on the data points on the first sensor and the data points on the second sensor, the same center point can be determined.

[0020] In another implementation of the method according to the first aspect, the first sensor and / or the second sensor includes one or more of a conventional camera, a stereo camera, a radar, and a lidar.

[0021] In other embodiments, the first sensor and / or the second sensor may include any other sensor modality that provides data suitable for performing the method thereon.

[0022] In another implementation of the method according to the first aspect, the one or more parameters of the calibration include a yaw parameter and a pitch parameter, and the method also includes filtering the second data based on a filter area, the filter area having an area width around a center line from the sensor toward a distant point, wherein the distant point has lateral and / or vertical coordinate components corresponding to the lateral and / or vertical coordinate components of the position of the sensor.

[0023] In other words, in this embodiment, the filter area can be a rectangular area in front of the vehicle (seen from above). Experiments have shown that this area is particularly useful as a filter area. The filter area herein refers to an area where data points are not filtered out, that is, the data points are retained and used for another calibration process.

[0024] Preferably, the zone width is determined based on a predetermined table assigning predetermined widths to given pairs of first and second sensors and / or the zone width is determined based on the position and / or orientation of the sensors.

[0025] In another implementation of the method according to the first aspect, the position of the distant point in the coordinates of the reference sensor is determined as:

[0026] -p ref_sensor =Tp cal_sensor ,

[0027] -p ref_sensor =K ref_sensor Tp cal_sensor ,

[0028] - as well as

[0029] - Among them, p cal_sensor is the position of the distant point in the coordinates of the calibration sensor, p ref_sensor is the position of the distant point in the coordinates of the reference sensor, T is the matrix transformation between the coordinate system of the calibration sensor and the coordinate system of the reference sensor, and K cal_sensor and K ref_sensor is the corresponding camera matrix transformation.

[0030] K cal_sensor and K ref_sensorThis can be the identity matrix. In general, the matrix used depends on the type of sensor used.

[0031] In another implementation of the method according to the first aspect, the one or more parameters of the calibration include at least one roll parameter, and the method also includes filtering the second data based on a filter area, the filter area having a fixed width around a center line from the sensor toward a distant point, the distant point being laterally moved from the sensor position by a lateral movement in the coordinates of the sensor.

[0032] In another implementation of the method according to the first aspect, filtering the second data includes filtering based on a filter area, wherein the filter area is adjusted based on a steering direction angle of the vehicle.

[0033] In another implementation of the method according to the first aspect, determining the lateral positioning of the filter area includes estimating the sine of the steering direction angle, wherein preferably, the estimation includes multiplying the sine of the steering direction angle by a sensitivity constant, preferably also multiplying by a constant proportional to the focal length of the camera device sensor.

[0034] In a further implementation of the method according to the first aspect, the method is repeatedly performed using a plurality of candidate a priori calibrations between the first sensor and the second sensor, and wherein preferably the method comprises an additional step of selecting a preferred a priori calibration from the plurality of candidate a priori calibrations.

[0035] In a further implementation of the method according to the first aspect, determining one or more parameters of the calibration between the first sensor and the second sensor comprises:

[0036] - determining a pairing between a point in the first data and a point in the second data, and

[0037] - Determining one or more parameters such that the error between paired points is minimized.

[0038] In another implementation of the method according to the first aspect, calibrating the first sensor and the second sensor also includes calibrating the third sensor with the first sensor and the second sensor, and determining one or more parameters of the calibration includes minimizing errors in point pairs between the first sensor and the second sensor, point pairs between the second sensor and the third sensor, and point pairs between the third sensor and the first sensor.

[0039] Preferably, the minimization of the errors of the point pairs comprises minimizing a weighted sum of the errors of the point pairs of the sensor pairs.

[0040] In another implementation of the method according to the first aspect, the method also includes verifying the a priori calibration, wherein the verification includes comparing one or more parameters of the determined calibration with one or more corresponding parameters of the a priori calibration, and verifying the calibration if the difference between the determined calibration and the a priori calibration is less than a predetermined threshold.

[0041] In a further implementation of the method according to the first aspect, the verifying comprises, for each of a plurality of sensors including the first sensor and the second sensor:

[0042] - using the determined calibration parameters to determine a first center point of the detected object when projected onto the coordinates of another sensor of the plurality of sensors,

[0043] - using the a priori calibrated parameters to determine a second center point of the detected object when projected onto the coordinates of another sensor of the plurality of sensors, and

[0044] - determining a distance between the first center point and the second center point, wherein preferably if the distance is less than a predetermined threshold, verifying a calibration between the sensor and another sensor of the plurality of sensors.

[0045] A second aspect of the present invention provides a calibration system for calibrating a first sensor and a second sensor of a vehicle, the calibration unit comprising:

[0046] - an obtaining unit for obtaining first data from the first sensor and second data from the second sensor,

[0047] - a filtering unit for filtering at least the second data based on positions of data points of the second data, and

[0048] A determination unit for determining one or more parameters of a calibration between the first sensor and the second sensor based on the first data and the filtered second data.

[0049] The calibration unit may be a calibration system.

[0050] Another aspect of the present invention relates to a computer-readable storage medium storing program code, the program code comprising instructions, which when executed by a processor, perform one of the implementations of the first aspect or the method of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical features of the embodiments of the present invention, the following briefly introduces the drawings provided for describing the embodiments. The drawings in the following description are only some embodiments of the present invention, and these embodiments can be modified without departing from the scope of the present invention as defined in the claims.

[0052] Figure 1 is a schematic illustration of a vehicle detected as an object in front of a car according to an embodiment of the present invention;

[0053] Figure 2A is a schematic illustration of the field of view of a vehicle and its two sensors (radar and camera) according to an embodiment of the present invention;

[0054] Figure 2B is a schematic illustration of a vehicle, a field of view and a basic filter area of ​​a sensor of the vehicle, and a shifted filter area based on a steering angle of the vehicle according to an embodiment of the present invention; and

[0055] Figure 3A and 3B is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The foregoing description is only an implementation of the present invention, and the scope of the present invention is not limited thereto. Any changes or substitutions can be easily made by those skilled in the art. Therefore, the protection scope of the present invention should be subject to the protection scope of the attached claims.

[0057] From the perspective of autonomous driving, multiple sensors (e.g., multiple cameras) are important for improving road safety. However, processing data from multiple sensors on a vehicle requires both accurate external calibration and accurate internal calibration for each camera. Accurate internal calibration of the cameras may include an optimal set of parameters of a camera projection model that relates 2D image points to 3D scene points; these optimal parameters correspond to a minimum reprojection error. Instead, in the following, we will refer to the task of external calibration. Accurate external calibration may correspond to an accurate camera pose on the vehicle relative to a reference frame. Accurate calibration allows feature points from one sensor to be reprojected into another sensor with a low reprojection error.

[0058] During the automatic external calibration process, hereinafter referred to only as calibration, the following data may be used:

[0059] • Conventional camera frame (2D) or depth image created by a stereo camera (3D).

[0060] ●Radar point cloud (2D / 3D)

[0061] ●LiDAR point cloud (3D)

[0062] • The sensor to be calibrated may provide additional data, for example, velocity, intensity, cross-sectional dimensions and / or confidence of detection. These may be used to assist the calibration process.

[0063] For sensors that implement tracking, these tracks can also be used to assist in the calibration process. (Trajectories can also show additional properties, such as physical size, confidence, etc.)

[0064] An important step before calibration is the pairing of the above detections between sensors. If the format of the detections varies between any sensor pair, first, the format of the detections is converted to the same domain, - in fact, to a domain that supports the sensor pairing. These common domains and corresponding conversions can be one or more of the following:

[0065] ● Conventional camera frame + point cloud: The points of the cloud are projected onto the camera's image space.

[0066] ●Point cloud + point cloud with tracking information: On the regular point cloud, object detection is performed. These objects can then be paired with tracks.

[0067] ●Point cloud + point cloud: Based on object detection, similar to the previous point. Note also that since the density of these point clouds may vary greatly, a one-to-one pairing of these point clouds may not be an option.

[0068] LiDAR devices can detect tens of thousands of points with each measurement, while radar devices can detect only tens to hundreds of points with each of their measurements, which complicates one-to-one pairing of such data.

[0069] Example of processing input

[0070] In the first embodiment, each detected object is represented by a single point. For calibration of a single camera device and a single radar device, the following operations may be performed:

[0071] For processing the camera frame, different (prior art) image processing and / or object detection methods can be used. For this example, let the objects to be detected be the cars visible on the camera frame. We can then use a trained neural network for detecting these objects, providing their positions in the image space.

[0072] For processing radar frames, we can use several methods, just as for camera frames. For example, using the relative speed provided by most radars, we can select which objects are moving and which are stationary. In addition, the point cloud can be clustered (several algorithms exist for this purpose), and then the center point of the objects found in this way can be easily set.

[0073] Figure 1 It shows how a sensor 100 (eg, a camera) sees a street 120 with vehicles 122 , 124 , and 126 in front of a vehicle (not shown) carrying the sensor 100 .

[0074] This method can be implemented differently when using different sensors: Using a radar device capable of tracking objects, the clustering step can be omitted since we can get the required positions from the device itself. Using e.g. a LiDAR instead of a camera, the LiDAR can just cluster the LiDAR point cloud as the radar clusters the radar point cloud.

[0075] In general, for each of the detected objects, a vehicle 122 , 124 , 126 , a bounding box 132 , 134 , 136 , and a center point of the bounding box 132 , 134 , 136 may be identified herein.

[0076] exist Figure 1 In FIG. 1 , the crosses 142, 144, 146 in the car indicate the centers of objects detected on the camera frame using the neural network. The circles 152, 154, 156 indicate the centers of clusters detected on the radar frame projected onto the image. As can be seen in FIG. Figure 1 As can be seen in FIG. 1 , the centers 142 , 144 , 146 of the objects detected on the camera frame are not perfectly aligned with the centers of the clusters detected on the radar frame.

[0077] Data pairing

[0078] An important aspect of the described calibration method is the way the data are paired. This can be a difficult task, since, for example, the a priori calibration used to convert the data of the paired sensors to the same domain can be severely corrupted. For this step, a method is needed that can work for all possible combinations of sensor pairings.

[0079] A simple way to pair is to choose the closest (spatially, using some criteria) point in the sensor. However, this method is highly error prone and requires the use of more sophisticated methods. Beyond this point, the literature does not provide any further guidance.

[0080] Using some filtering methods, enough points can be eliminated to minimize the chance of getting false pairs. The filters are:

[0081] ○ Simple - We only need to work with one point (the center point of the detected object) for each detected object. Filtering of this data (e.g., filtering by the location of this data) requires less computing power.

[0082] ○ Robust—filtering can be implemented on all sensors.

[0083] It should be understood that, for example, both data from a first sensor and data from a second sensor may be filtered, and in this case, different filtering rules may be applied to data from different sensors. At least one of the filtering methods applied is based on the location of the data, i.e., the decision of which data point to keep and which data point to filter out is based on the location of the data point in the data.

[0084] A priori calibration can contain significant errors. The goal of the proposed method consists in eliminating these errors. If there are no errors or only minimal errors, the distances of the point pairs are minimal, so the result of the calibration will be highly similar to the prior. Therefore, the method can also be used for the verification of existing calibration parameters.

[0085] The selection of sensor pairs can be made by the user or by software. The pair can be any combination of existing cameras, radar devices, and lidar devices, as long as they share a common field of view.

[0086] In general: When calibrating a sensor's yaw and pitch, choose the center of its field of view. When calibrating a sensor's roll, choose the periphery of its field of view. Also, for the second sensor, the chosen field of view depends on the field of view of the first sensor and the prior calibration (e.g. one sensor can look forwards while the other can look to the side).

[0087] For yaw and pitch calibration, the following steps can be used:

[0088] Select a virtual point in front of the sensor (P cal_sensor ): X meters from the sensor in the longitudinal direction. (In our case, X=50.) In the horizontal and vertical directions, the position of the point does not differ from the position of the sensor. (The point is defined in the coordinates of the sensor.)

[0089] The width of the filter area can be selected by using a predefined list containing each sensor. (In our case, the width is about 3m to 4m, which is equal to the lane width.)

[0090] ■The values ​​in this list are user-defined. The values ​​in this list should be selected taking into account the location and orientation of the sensor, the possible environment, and the objects to be detected.

[0091] ■The list should contain all possible combinations of sensor pairings.

[0092] Find out the width of the filter area and set its center to the previously defined virtual point (P cal_sensor), the region is well defined. This region can be transformed between sensors using the a priori calibration of the sensors. (For example, let T be the matrix transformation between the sensor coordinate systems). Furthermore, let us assume that one of the sensors is a camera, in which case we must also apply the K camera matrix transformation to project the 3D points onto image space. Using these, we can write the following equation: P ref_sensor =(K*)T*P cal_sensor .

[0093] For the calibration of the roll, the same steps as before are used, except that the position of the virtual point - its lateral distance is not zero, but has a value of an object on the surrounding area of ​​the field of view that can be optimally selected. For the forward-looking radar, this value can be selected as Y = + / - x meters, where x is between 3 meters and 12 meters, preferably between 4 meters and 8 meters. These values ​​have proven to be beneficial because they correspond approximately to the left and right lanes.

[0094] Figure 2A 2 is a schematic illustration of the field of view of a vehicle 200 and its two sensors (radar 210 and camera 220). Specifically, radar 210 has a field of view 212, and camera 220 has a field of view 222 that partially overlaps with radar field of view 212. The left and right boundaries 234a, 234b of basic filter region 240 are defined parallel to a centerline defined by the longitudinal middle of the vehicle (at Figure 2B In the horizontal direction, these boundaries 234a, 234b are equidistant from the center line.

[0095] The method can be performed with minimal reliance on a priori calibration, as long as one of the paired sensors is significantly different from the direction of vehicle travel, while the other sensor is well aligned with that direction of travel. Errors in the a priori calibration are realized as errors in the projected filter area. This reliance on a priori calibration can be mitigated in several ways:

[0096] o If the first result is not satisfactory, iteratively performing the method will lead to a better solution.

[0097] ○ Performing the filtering multiple times with different modified a priori calibrations can compensate for errors in the a priori calibration. (As the number of times the filtering is performed increases, the execution time will also increase, but since the filtering algorithm is lightweight (see above), the entire process will remain lightweight.)

[0098] It is also possible to modify the selected filter area based on the steering wheel angle. This can further increase the chances of successful pairing, since on a curvy road other cars may not be within the filter area if the filter area is not modified when turning. The filter area can be modified to:

[0099] ○ In case of 3D detection: Y new =Y old + sin(steering angle)*X*c (Here, c is a preset constant that modifies the sensitivity of the steering effect.)

[0100] ○ In the case of image space detection: Y new =Y old + sin(steering angle)*X*c*f x / res x (Here, f x is the focal length of the camera, res x is the width of the camera device in pixels. )

[0101] Figure 2B 2 is a schematic illustration of a car 200, a field of view 234 of a sensor 230 of the car, a basic filter region 240, and a shifted filter region 242 based on the steering angle of the car. Specifically, it can be seen that the basic filter region 240 corresponding to the filter region when the car is driving in a straight line has boundaries 234a, 234b.

[0102] When the vehicle turns to the left, the filter area is also shifted to the left, so that a new filter area 242 is obtained, which has boundaries 236a, 236b shifted to the left from boundaries 234a, 234b of the basic filter area.

[0103] In one embodiment, the shift of the boundary is proportional to the steering angle or the sine of the steering angle or another function of the steering angle. The width of the shifted filter area can be the same as the width of the basic filter area, or in other embodiments, the width of the shifted filter area can also vary based on a function of the steering angle.

[0104] If you can Figure 2B As can be seen in FIG. 1 , the leftward shift of the filter area has the advantage that the filter area is shifted in the direction the vehicle is turning. Therefore, it is more likely that the lead vehicle 202 falls within the filter area. Although filtering significantly reduces the number of data points that fall within the overlap of the filter area and the field of view, it is very likely that the lead vehicle falls within the overlap.

[0105] In other words, in one embodiment, a car driving in front of the vehicle with the sensor is outside the center area as seen on the radar frame if the area is not offset by the steering angle. Therefore, using the steering angle to modify the filter area can shorten the data collection time without changing the quality of the results.

[0106] If you can Figure 2B As can be seen in FIG. 2 , the filter area typically only partially overlaps the field of view 232 of the sensor.

[0107] Calculation results

[0108] To obtain the extrinsic calibration of the sensor in question, we compute the transformation for which the error between its paired points is minimized. A good choice for achieving this is to use optimization, which can iteratively minimize the error between paired points to arrive at a transformation state that represents the true extrinsic calibration well enough.

[0109] The point pairing method described above is fast and robust, and it can be used in the case of any of the mentioned sensor types. In addition, due to the point pairing method, more than two sensors can also be calibrated together at the same time, which further increases the stability of the method. This is also called bundle adjustment. For our case, a possible formalization of the method is as follows. Assume that there are three sensors (A, B and C). Using conventional optimization, only the errors between point pairs such as sensor pair AB are minimized. By using bundle adjustment, we can minimize the errors between point pairs such as sensor pairs AB, AC and BC at the same time, which can produce better overall results.

[0110] Similarly for Figure 1 : In the figure, we can see that the detected objects on the camera frame are marked with small crosses, and the detected objects on the radar frame are marked with circles. In addition, in practice, there may be additional points (in Figure 1 The bundle adjustment simultaneously minimizes the error between point pairs for all possible sensor combinations, thus effectively reducing the overall error.

[0111] During bundle adjustment, the error metric between any two sensor pairs can be different. Such a decision is made by the user.

[0112] The use of bundle adjustment is not necessary for implementing the method, conventional optimization or any other method to achieve the same goal may be applied to solve the problem. In a preferred embodiment, bundle adjustment is used due to its obvious advantages and its universality.

[0113] verify

[0114] The basis for the validation is the calibration process described above. If the calibration of the sensor gives the same (or nearly the same) extrinsic parameters as the a priori calibration, the a priori calibration is considered to still be valid.

[0115] Although it may seem straightforward at this point, a trivial comparison of the a priori calibration extrinsics and the generated calibration extrinsics is usually not the best solution. Instead, the following may be preferred: for each sensor calibrated to another sensor, we calculate the center point of its detection when projected onto the space of the other sensor using both the a priori extrinsics just found and the generated extrinsics. Then, for both the a priori case and the generated case, calculate the errors between these points and their pairs. This may have been found during the first and last steps of the optimization. The metric used to calculate the errors is free to use and is decided by the user. The errors can be compared, for example, by taking the error ratio and setting it against some threshold.

[0116] By repeating this method in a continuous manner, we can achieve a continuous verification process of the extrinsic parameters of the sensor.

[0117] Before validation is possible, an a priori calibration is obtained using the calibration described above.

[0118] Figure 3A and Figure 3B is a further detailed flow chart of a method for initializing 300, collecting data 310, calculating 320 results, and validating 340 original parameters (also referred to as a priori parameters).

[0119] In detail, initialization 300 includes the following steps:

[0120] In a first step 302, raw extrinsic parameters are initialized. This may involve determining an a priori calibration based on other sources (eg information about the positioning of the sensor).

[0121] Then, in step 304, one or more sensors to be calibrated are selected and sensor pairings are determined. In step 306, a basic filter area is calculated. The basic filter area can be based on yaw-pitch and roll. Subsequently, in 308, three proposed filter areas are determined, and the following steps of collecting data and calculating results are performed for the three proposed filter areas respectively. Determining the proposed filter area can include determining one or more parameters of the proposed filter area, but it should be understood that additional parameters can be adjusted during data acquisition.

[0122] In an embodiment, the width and the centerline of the filter area are determined during these steps (the width and the centerline of the filter area may subsequently be shifted by the steering direction of the vehicle).

[0123] In a preferred embodiment, the centerline is known on the sensor that should be calibrated, however, it is unknown on the reference sensor. Therefore, the centerline must be transformed into the coordinate system of the reference sensor. Since the exact external parameters are unknown, this step (may) introduce errors, so an optimization can be performed by applying multiple transformations with slightly different parameters, for example based on different methods of determining the transformation parameters or based on (e.g. random) changes introduced to the parameters. Based on multiple different transformations, multiple filter areas can be determined, and the filter area that produces the best results can be used at the end of the optimization.

[0124] Data collection 310 may be performed when a vehicle (or preferably a fleet) is driving on a road for real-world data acquisition. However, it should be understood that the proposed method may also be applied in a virtual scenario where a virtual vehicle is driving in a virtual world (and acquiring virtual data from a simulated environment).

[0125] Data collection 310 also includes the following steps:

[0126] First, in step 312a, data for yaw-pitch calibration is collected. In parallel, in step 312b, data for roll calibration is collected.

[0127] In steps 314a, 314b, the filter area is moved based on the current steering angle of the vehicle on which the one or more sensors are mounted. The filter area of ​​one or more or all sensors may be moved based on the steering angle. In addition to the steering angle, additional information such as the speed of the vehicle may also be used to move or otherwise adjust the filter area.

[0128] In steps 316a, 316b, corresponding pairs are identified in the filtered data.

[0129] Result calculation 320 includes the following steps:

[0130] First, for all three collected data sets, the cost function is estimated and optimized in steps 322a, 322b, and 322c. Even after extensive optimization, small errors remain, and these remaining errors are left in steps 324a, 324b, 324c. Subsequently, in step 326, the errors can be compared and the best result selected.

[0131] The final results of the calibrated extrinsic parameters 330 and the original extrinsic parameters 302 may then be provided to a validation 340 .

[0132] Verification 340 includes the following steps:

[0133] In step 342, the paired points are reprojected, and the error is calculated in step 344. Finally, a validation metric is applied to estimate whether the prior parameters are correct. For example, if the error calculated in step 344 is less than a predetermined threshold, the prior parameters can be considered correct. Alternatively, the method can be performed on several prior parameters, and the prior parameters that produce the smallest calculated error are determined to be the correct parameters.

Claims

1. A method for calibrating a first sensor and a second sensor of a vehicle (122, 124, 126), the method comprising: - obtaining first data from said first sensor and second data from said second sensor, - filtering at least said second data based on positions of data points of said second data, and - Determining one or more parameters of a calibration between the first sensor and the second sensor based on the first data and the filtered second data (302, 330). 2 . The method of claim 1 , further comprising converting the first data and / or the second data into a common domain using an a priori calibration.

3. The method according to claim 1 or 2, further comprising the steps of detecting a first object in the first data and / or detecting a second object in the filtered second data, and performing the determination of the one or more parameters (302, 330) based on the first object and / or the second object, preferably, wherein: The determining of the one or more parameters (302, 330) is based on a center point of the first object and / or the second object.

4. The method according to one of the preceding claims, wherein: The first sensor and / or the second sensor include one or more of a conventional camera, a stereo camera, a radar (210), and a laser radar.

5. The method according to one of the preceding claims, wherein: The one or more parameters (302, 330) of the calibration include a yaw parameter and a pitch parameter (302, 330), and the method further includes filtering the second data based on a filter area, the filter area having a region width around a center line from the sensor (100, 230) toward a distant point, wherein the distant point has a lateral coordinate component and / or a vertical coordinate component corresponding to the lateral coordinate component and / or the vertical coordinate component of the position of the sensor (100, 230).

6. The method according to claim 5, wherein: - determining said area width based on a predetermined table assigning predetermined widths to given pairs of first and second sensors, and / or - determining the zone width based on the position and / or orientation of the sensor (100, 230).

7. The method according to claim 5 or 6, wherein: The position of the distant point in the coordinates of the reference sensor is determined as: -p ref_sensor =T P cal_sensor , -P ref_sensor =K ref_sensor T P cal_sensor , _ as well as _ Among them, p cal_sensor is the position of the distant point in the coordinates of the calibration sensor, p ref_sensor is the position of the distant point in the coordinates of the reference sensor, T is the matrix transformation between the coordinate system of the calibration sensor and the coordinate system of the reference sensor, and K cal_sensor and K ref_sensor is the corresponding camera matrix transformation.

8. The method according to one of the preceding claims, wherein: The one or more parameters (302, 330) of the calibration include at least one roll parameter, and the method further includes filtering the second data based on a filter region having a fixed width around a centerline from the sensor (100, 230) toward a distant point, the distant point being laterally displaced from a sensor position in coordinates of the sensor (100, 230) by a lateral displacement.

9. The method according to one of the preceding claims, wherein: The filtering of the second data comprises filtering based on a filtering area, wherein the filtering area is adjusted based on a steering direction angle of the vehicle (122, 124, 126), preferably wherein determining the lateral positioning of the filtering area comprises estimating the sine value of the steering direction angle, wherein preferably the estimation comprises multiplying the sine value of the steering direction angle by a sensitivity constant, preferably also by a constant proportional to the focal length of the camera device sensor.

10. The method according to one of the preceding claims, wherein: The method is repeatedly performed using a plurality of candidate a-priori calibrations between the first sensor and the second sensor, and wherein preferably the method comprises the additional step of selecting a preferred a-priori calibration from the plurality of candidate a-priori calibrations.

11. The method according to one of the preceding claims, wherein: The determining one or more parameters of a calibration between the first sensor and the second sensor (302, 330) includes: - determining a pairing between a point in the first data and a point in the second data, and - determining the one or more parameters (302, 330) so as to minimize the error between the paired points, Preferably, wherein calibrating the first sensor and the second sensor further comprises calibrating a third sensor using the first sensor and the second sensor, and wherein determining one or more parameters of the calibration comprises minimizing errors of point pairs between the first sensor and the second sensor, point pairs between the second sensor and the third sensor, and point pairs between the third sensor and the first sensor, wherein preferably, minimizing the errors of the point pairs comprises minimizing the weighted sum of the errors of the point pairs of the sensor pairs.

12. The method according to one of the preceding claims, further comprising verifying (340) the determined calibration, wherein The verification (340) includes comparing the one or more parameters (302, 330) of the determined calibration with one or more corresponding parameters (302, 330) of a priori calibration, and verifying (340) the a priori calibration if a difference between the determined calibration and the a priori calibration is less than a predetermined threshold.

13. The method according to claim 12, wherein: The verification (340) includes: for each of a plurality of sensors (100, 230) including the first sensor and the second sensor, - using the determined calibration parameters (302, 330) to determine a first center point of the detected object when projected onto the coordinates of another sensor of the plurality of sensors (100, 230), - using the a priori calibrated parameters to determine a second center point of the detected object when projected onto the coordinates of another sensor of the plurality of sensors, and - determining the distance between said first centre point and said second centre point, Preferably, if the distance is less than a predetermined threshold, calibration between a sensor (100, 230) and another sensor in the plurality of sensors (100, 230) is verified.

14. A calibration system for calibrating a first sensor and a second sensor of a vehicle (122, 124, 126), the calibration unit comprising: - an obtaining unit, the obtaining unit being configured to obtain first data from the first sensor and second data from the second sensor, - a filtering unit configured to filter at least the second data based on positions of data points of the second data, and A determination unit for determining one or more parameters of a calibration between the first sensor and the second sensor based on the first data and the filtered second data (302, 330).

15. A computer-readable storage medium storing program code, the program code comprising instructions which, when executed by a processor, perform the method of one of claims 1 to 13.

Images