Methods for validating environmental sensing sensors of a vehicle and for validating a vehicle equipped with environmental sensing sensors

Abstract

Method for validating a plurality of environment detection sensors rigidly connected to a vehicle (1), which are configured to detect a relative velocity (V1 to V7, Vn) of at least one object in the vicinity of the vehicle (1) relative to a sensor coordinate system (S1 to S7, Sn), and - in an extrinsic calibration for each sensor coordinate system (S1 to S7, Sn) a homogeneous coordinate transformation is determined to convert coordinates of the sensor coordinate system (S1 to S7, Sn) into coordinates of a vehicle coordinate system (V) rigidly connected to the vehicle (1), - based on homogeneous coordinate transformations ◯ For each relative velocity (V1 to V7, Vn), an object velocity (X1 to X7, Xn) related to the vehicle coordinate system (V) is determined and / or ◯ from the majority of the relative velocities (V1 to V7, Vn) at least one parameter of a motion model of the vehicle (1) is determined, characterized in that - a decalibrated state (C0) is assigned to the majority of environmental sensing sensors when the object velocities (X1 to X7, Xn) deviate from each other and / or from the motion model of the vehicle (1) by more than a predetermined amount and - Deviations of the object velocities (X1 to X7, Xn) are determined as pairwise vector differences and the maximum pairwise vector difference in magnitude and / or direction is compared with a predetermined magnitude difference or a predetermined angular difference.

Description

[0001] The invention relates to a method for validating an extrinsic calibration of a plurality of environmental sensors rigidly connected to a vehicle, each of which is configured to detect a relative velocity related to a sensor coordinate system. The invention further relates to a vehicle having such environmental sensors and having at least one control unit.

[0002] DE 10 2005 037 094 B3 discloses a method for calibrating a distance measurement sensor with two sensor channels and a beam pattern that is lobed. The method comprises the following steps: - Emitting a radiation beam of a sensor channel onto a calibration surface by means of the sensor, - Capturing the radiation lobe with a camera of a video system, - Determining a first representative value of the radiation lobe in image coordinates of the camera, - Changing the position of the calibration surface in relation to the sensor and the camera, - Capturing the radiation lobe with the camera, - Determining a second representative value of the radiation lobe in image coordinates of the camera, - Repeat the previous steps for another sensor channel, - Modeling of beam axes of the emitted radiation beams as straight lines in sensor coordinates, - Transforming the modeled straight lines from sensor coordinates into image coordinates of the camera, - Comparing the straight lines with the determined representative values ​​in image coordinates for each sensor channel, - Determination of a calibration function to compensate for a possible deviation of the modeled straight line from at least several representative values.

[0003] Furthermore, WO 2019 / 154536 A1 describes a method for object detection using proximity sensors. Lidar and radar data are transformed into a common coordinate system. Object speed is then estimated using clusters of Lidar and radar points.

[0004] From DE 10 2012 018 012 A1 a method for determining the position of objects is known in which a correction of the position determination is carried out on the basis of the movement path of a stationary object and the angular deviation of this movement path from that of the vehicle.

[0005] DE 10 2016 106 299 A1 discloses a method in which the position, orientation, and speed of a remote vehicle are determined by fusing data from two environmental sensors. Data from the positions of the vehicle's wheels is used.

[0006] The invention is based on the object of providing a novel method for validating environmental detection sensors of a vehicle. The invention is further based on the object of providing a vehicle that is configured to validate such environmental detection sensors.

[0007] With regard to the method, the object is achieved according to the invention by a method which has the features specified in claim 1.

[0008] With regard to the vehicle, the object is achieved according to the invention by a vehicle which has the features specified in claim 4.

[0009] Advantageous embodiments of the invention are the subject of the subclaims.

[0010] In a method for validating a plurality of environmental detection sensors rigidly connected to a vehicle, each of which is configured to detect a relative velocity of at least one object in the vehicle's surroundings, each relative to an associated sensor coordinate system, according to the invention, a homogeneous coordinate transformation is determined for each sensor coordinate system in an extrinsic calibration to convert coordinates of the sensor coordinate system into coordinates of a vehicle coordinate system rigidly connected to the vehicle. The relative velocity indicates, in terms of magnitude and direction, the speed of movement of an object in the vehicle's surroundings relative to the respective sensor coordinate system.

[0011] Using the homogeneous coordinate transformation assigned to the respective sensor coordinate system, an object speed relative to the vehicle coordinate system is determined for each relative speed detected by an environment detection sensor. The object speed indicates, in magnitude and direction, the speed of movement of an object in the vehicle's surroundings relative to the vehicle coordinate system.

[0012] Alternatively or in addition to determining the object speeds, at least one parameter of a vehicle's motion model is determined from the plurality of relative speeds. A vehicle's motion model can, for example, be specified as the instantaneous vehicle speed in terms of magnitude and direction. However, motion models are also possible in which other or additional parameters, such as at least one angular velocity or a radius of curvature of a trajectory traveled by the vehicle, are recorded.

[0013] A decalibrated state is assigned from the plurality of environmental detection sensors if the object speeds determined with reference to the vehicle coordinate system deviate from each other by more than a predetermined amount.

[0014] A decalibrated state is understood here and below as a change in the pose (i.e. the position and / or orientation) of at least one sensor coordinate system relative to at least one other sensor coordinate system compared to the pose recorded in the extrinsic calibration, whereby this change in the pose means that the vehicle speed can no longer be validly determined from the relative speeds.

[0015] Alternatively or additionally, a decalibrated state is assigned if the object speeds deviate by more than a predetermined amount from the speeds which the objects in the vehicle environment have based on a movement model of the vehicle determined with at least one parameter from the relative speeds.

[0016] Furthermore, deviations between object velocities assigned to different sensor coordinate systems are determined pairwise as a vector difference. The maximum pairwise vector difference in magnitude and / or direction is compared with a predetermined magnitude difference (with regard to the magnitude of the pairwise vector difference) or with a predetermined angular difference (with regard to the direction of the pairwise vector difference). A decalibrated state is assigned to the plurality of environment detection sensors if at least one pairwise vector difference exceeds the predetermined magnitude difference and / or angular difference.

[0017] One advantage of the method is that extrinsic calibration can be validated when relative speeds of different objects are detected by different environmental detection sensors. In particular, validation of extrinsic calibration is also possible when the areas of the vehicle's environment covered by different environmental detection sensors do not overlap. Furthermore, validation is possible without identifying objects, i.e., without recognizing them as having been detected consistently by different environmental detection sensors.

[0018] Furthermore, validation according to the method according to the invention is possible without access to a digital map containing objects that can potentially be detected by an environmental detection sensor. This makes it possible to perform validation continuously and in essentially any environment, especially in unmapped environments.

[0019] The method according to the invention thus enables a more reliable and simpler validation of a plurality of environmental detection sensors than methods known from the prior art.

[0020] A further advantage of the invention is that a decalibrated state can be determined particularly easily.

[0021] In one embodiment of the method, the vehicle's motion model is determined as a vehicle speed relative to the vehicle coordinate system, based on magnitude and direction. From the vehicle speed, a target relative speed is determined for each sensor coordinate system according to the respective assigned homogeneous coordinate transformation. At least one target relative speed is compared with the relative speed recorded for the respective sensor coordinate system, based on magnitude and / or direction.

[0022] An advantage of this embodiment is that a decalibrated state can be determined particularly reliably. In particular, deviations of several sensor coordinate systems that are very similar in magnitude and direction can be reliably detected.

[0023] In one embodiment, the confidence level of the assignment of the decalibrated state is determined statistically from the majority of relative velocities. For example, confidence level can be determined from the relative proportion of sensor coordinate systems whose assigned object velocities do not differ from each other or differ only slightly, i.e., by less than the predetermined amount.

[0024] Alternatively or additionally, a calibrated state can be assigned if the object speeds do not deviate from each other and from the vehicle's motion model or deviate only slightly, i.e. by less than the predetermined amount.

[0025] A calibrated state is understood here and in the following to mean that the poses of all sensor coordinate systems do not deviate from the pose recorded in the extrinsic calibration or only deviate to such an extent that the vehicle speed can still be validly determined from the relative speeds.

[0026] In the same way as already explained for the confidence of the assignment of the decalibrated state, the confidence of the assignment of the calibrated state is also determined statistically.

[0027] An advantage of this embodiment of the method is that, in the case of unreliable measurements from individual environmental detection sensors, for example, when a vehicle ahead is detected by at least one environmental detection sensor and a stationary object is simultaneously detected by at least one other environmental detection sensor, incorrect state assignments can be detected. This enables more robust and fault-tolerant validation.

[0028] In a vehicle comprising at least one computing unit and a plurality of environmental detection sensors, which are configured to detect a relative speed of at least one object detected in the environment of the vehicle, related to a sensor coordinate system of the respective environmental detection sensor, the environmental detection sensors and the at least one computing unit are configured according to the invention to carry out the described method for validating the plurality of environmental detection sensors.

[0029] Such a vehicle offers the advantage that decalibration of the environmental detection sensors can be detected particularly easily and reliably, and errors in vehicle functions that are based on the evaluation of measurement data from these environmental detection sensors, such as incorrect or missing warnings to the driver or incorrect vehicle control, can be detected or prevented. This enables a more reliable and safer vehicle.

[0030] In a particularly space- and cost-saving embodiment, the at least one computing unit is designed as a control unit.

[0031] Embodiments of the invention are explained in more detail below with reference to drawings.

[0032] Showing: Fig. 1 schematically shows a vehicle with speed measurement sensors in an arrangement for extrinsic calibration, Fig. 2 schematically shows relative velocities and object velocities with the sensor pose unchanged compared to the extrinsic calibration, Fig. 3 schematically shows a vehicle with sensors in a different sensor pose compared to the extrinsic calibration, Fig. 4 schematically shows relative velocities and object velocities when the sensor pose is changed compared to the extrinsic calibration, Fig. 5 schematically shows the flow chart of a procedure for distinguishing between a calibrated and a decalibrated state, Fig. 6 schematically shows a vehicle with sensors in a sensor pose unchanged compared to the extrinsic calibration during cornering and Fig. 7 schematically shows a vehicle with sensors in a sensor pose unchanged compared to the extrinsic calibration during cornering.

[0033] Corresponding parts are provided with the same reference numerals in all figures. Fig. Figure 1 shows a vehicle 1 equipped with seven sensors (not shown in detail). Each sensor is configured to measure the magnitude and direction of a relative velocity V1 to V7 relative to a sensor coordinate system S1 to S7.

[0034] Such sensors can be implemented, for example, as LIDAR or RADAR sensors or as time-of-flight (ToF) cameras. A relative velocity V1 to V7 can also be determined by recording the distance of a vehicle-independent, stationary object from the respective sensor coordinate system S1 to S7 using a camera at successive measurement times. A relative velocity V1 to V7 is calculated from the relative movement of the object in the sensor coordinate system S1 to S7, based on the difference between the measurement times.

[0035] To simplify the presentation, Fig. 1, a first to seventh relative velocities V1 to V7 are each represented in a two-dimensional Cartesian sensor coordinate system S1 to S7. However, sensors are also available with which a relative velocity V1 to V7 can be recorded as a three-dimensional vector quantity. The method described below can also be implemented without limitation for three-dimensionally recorded relative velocities.

[0036] The sensors are rigidly connected to each other via vehicle 1 and follow its movement. The positional relationship of the sensor coordinate systems S1 to S7 to each other as well as to a vehicle coordinate system V can thus be described using a homogeneous coordinate transformation. In particular, the second to fifth sensor coordinate systems S2 to S5 and the seventh sensor coordinate system S7 are rotated relative to each other and relative to the vehicle coordinate system V.

[0037] In a method known from the state of the art and referred to as extrinsic calibration, a homogeneous coordinate transformation is determined once for each of the sensor coordinate systems S1 to S7 and subsequently used to transform a relative speed V1 to V7 recorded with the respective sensor into the vehicle coordinate system V, as in Fig. 2 is presented in more detail.

[0038] Fig. 2 shows the first to seventh relative speeds V1 to V7, each detected by a sensor, as a two-dimensional vector quantity in the respectively assigned sensor coordinate system S1 to S7 during a uniform rectilinear movement of the vehicle 1. Due to their rotation relative to one another, the second to fifth and the seventh relative speeds V2 to V5, V7 have a different direction and, in some cases, also a different magnitude in the second to fifth and the seventh sensor coordinate system S2 to S5, S7.

[0039] By applying the sensor-related homogeneous coordinate transformation, an estimated object speed X1 to X7 is determined for each relative speed V1 to V7, which indicates the estimated speed of the respective sensor in terms of magnitude and direction relative to the vehicle coordinate system V.

[0040] If the sensor-related homogeneous coordinate transformations have been correctly determined in the extrinsic calibration and the sensor coordinate systems S1 to S7 are unchanged in their position relative to each other and to the vehicle 1, then, with a uniform linear movement of the vehicle 1, an object speed X1 to X7 with the same magnitude and direction relative to the vehicle coordinate system V is determined by applying the sensor-related homogeneous coordinate transformation to each of the relative speeds V1 to V7, as in Fig. 2 shown.

[0041] Fig. 3 shows the vehicle 1 with its sensor coordinate systems S1 to S7 assigned to the sensors not shown in detail. Fig. 1, the fourth sensor coordinate system S4 is changed compared to the state in which the extrinsic calibration was performed.

[0042] In particular, the fourth sensor coordinate system S4 is rotated by an angular offset α compared to an originally calibrated fourth sensor coordinate system S4', with which the extrinsic calibration was performed.

[0043] Accordingly, if the vehicle 1 is moved in the same way as by the relative velocities V1 to V7 according to the Fig. 2, the fourth relative speed V4 in the fourth sensor coordinate system S4 is also rotated, while the remaining relative speeds V1 to V3, V5 to V7 are different in magnitude and direction from the Fig. 2 remain unchanged.

[0044] Thus, the application of the sensor-related homogeneous coordinate transformations for these remaining relative velocities V1 to V3, V5 to V7 results in matching object velocities X1 to X3, X5 to X7 to each other and also to the movement of the vehicle 1, as in Fig. 4. The angular offset α of the fourth sensor coordinate system S4 compared to the extrinsic calibration, however, also causes an angular offset α of the fourth object velocity X4, which is determined by applying the homogeneous coordinate transformation related to the originally calibrated fourth sensor coordinate system S4'.

[0045] The invention is based on the finding that a positional deviation of a sensor coordinate system S4 compared to an original sensor coordinate system S4' at the time of the extrinsic calibration can be detected from the deviation of a single - in this case the fourth - object speed X4 compared to a plurality of other, mutually consistent object speeds X1 to X3, X5 to X7.

[0046] Fig. 5 explains the procedure for detecting such a position deviation in more detail.

[0047] For the first to n-th sensor coordinate systems S1 to Sn, a first to n-th relative velocity V1 to Vn is estimated from the respective associated sensor data D1 to Dn in a motion estimation step BSS. For example, a relative velocity V1 to Vn can be estimated from the spatial movement of an object in the sensor coordinate system S1 to Sn.

[0048] From the plurality of relative velocities V1 to Vn estimated in this way, a motion model of the vehicle 1 is parameterized in a subsequent parameterization step PS.

[0049] In addition to the estimated relative velocities V1 to Vn, each related to a sensor coordinate system S1 to Sn, extrinsic parameters P are included in the parameterization step PS. extwhich describe the position of the sensor coordinate systems S1 to Sn relative to the vehicle coordinate system V (and thus also their position relative to each other). For example, the extrinsic parameters P ext are provided as parameters of all homogeneous coordinate transformations that describe the pose (i.e., the offset and rotation) of a sensor coordinate system S1 to Sn relative to the vehicle coordinate system V. The extrinsic parameters P ext are determined in a prior extrinsic calibration using methods known from the state of the art.

[0050] The parameterized motion model of the vehicle 1 comprises, for example, a speed component along a longitudinal direction of the vehicle 1 when driving straight ahead. Optionally, the parameterized motion model comprises, for example, a further speed component along a transverse direction of the vehicle 1 arranged perpendicular to the longitudinal direction when cornering. Alternatively or additionally, a parameterized motion model when cornering can also comprise a radius of curvature of a vehicle trajectory K, as described below with reference to Fig. 6 and Fig. 7 will be explained later. Other vehicle kinematic parameters can also be included in the parameterized motion model.

[0051] In a subsequent transformation step TS, again carried out separately for each of the sensor coordinate systems S1 to Sn, an object velocity X1 to Xn is determined for each of the first to nth relative velocities V1 to Vn by applying the parameterized motion model. An object velocity X1 to Xn indicates, in terms of direction and magnitude, a velocity that the respective sensor has relative to the vehicle coordinate system V, matching the motion model determined in the parameterization step PS, if the respective sensor's pose relative to the vehicle coordinate system V remains unchanged compared to the extrinsic calibration.

[0052] In a subsequent decision step E, the object speeds X1 to Xn are compared with each other and / or for each of the sensor coordinate systems S1 to Sn the respectively determined object speed X1 to Xn is compared with the respectively determined relative speed.

[0053] In one embodiment, object speeds X1 to Xn that deviate particularly noticeably from the majority of the other determined object speeds X1 to Xn are identified as outliers. Methods for detecting outliers are known from the prior art. For example, a mean and a standard deviation can be determined from the total of the object speeds X1 to Xn. An object speed X1 to Xn can be identified as an outlier if it deviates from the mean by a multiple of the standard deviation.

[0054] If one or more such outliers are detected, a decalibrated state C0 is assigned as a result following the decision step E, which indicates that vehicle poses of vehicle 1, which are determined based on the sensor data D1 to Dn, are not trustworthy.

[0055] If no outlier is detected among the determined object speeds X1 to Xn, a calibrated state C1 is assigned as a result following the decision step E, which indicates that vehicle poses determined based on these sensor data D1 to Dn are still trustworthy.

[0056] This allows the trustworthiness (or reliability) of the determination of vehicle poses to be determined without requiring reference measurements from multiple sensor coordinate systems S1 to Sn with respect to a common reference object. In particular, it is not necessary to determine or ensure whether or that the same reference object is measured from multiple or even all sensor coordinate systems S1 to Sn. It is therefore not necessary to identify a reference object. In particular, it is also not necessary to record such reference objects in a digital map and to compare measurements in the sensor coordinate systems S1 to Sn with a digital map.

[0057] Alternatively or additionally, the mean square distance of the object velocities X1 to Xn from a mean (vector) object velocity can be determined as a measure of the agreement of the current positional relationships of the sensor coordinate systems S1 to Sn with those at the time of the extrinsic calibration.

[0058] If the mean square distance (or a similar distance measure for the object velocities X1 to Xn) exceeds a predetermined threshold, a decalibrated state C0 is assigned as a result following decision step E, indicating that the vehicle poses of vehicle 1 determined based on sensor data D1 to Dn are not trustworthy. Otherwise, a calibrated state C1 is assigned as a result following decision step E, indicating that the vehicle poses determined based on these sensor data D1 to Dn are still trustworthy.

[0059] An advantage of this embodiment is that the decalibration of several sensors can be determined more reliably than with outlier detection methods.

[0060] The distance measure can also be determined as a relative distance relative to an average object speed, for example, as a coefficient of variation of the magnitudes of the object speeds X1 to Xn. An advantage of this embodiment is that a more robust detection of a decalibrated state C0 is possible.

[0061] The Fig. 6 and Fig. 7 illustrate the method in its application to a vehicle 1 that is moved along a vehicle trajectory K that is formed as a circular segment.

[0062] Fig. 6 shows vehicle 1 of Fig. 1 during cornering with sensor coordinate systems S1 to S7 unchanged compared to the extrinsic calibration. By applying the extrinsic calibration for the fourth From the homogeneous coordinate transformation determined from the sensor coordinate system S4 to the measured fourth relative velocity V4, a fourth object velocity X4 is determined with respect to the vehicle coordinate system V.

[0063] Fig. Figure 7 shows a modified vehicle 1, in which the fourth sensor coordinate system S4 is aligned the same way but shifted relative to the calibrated fourth sensor coordinate system S4'. Such a shift does not cause a change in the measured fourth relative speed V4 when the vehicle 1 is traveling straight ahead, and thus also does not cause a deviation in the fourth object speed X4. In other words, a pure shift of a sensor coordinate system S4 cannot be detected by comparing the object speeds X1 to X7, which are determined by applying the homogeneous coordinate transformations, when traveling straight ahead.

[0064] If, however, vehicle 1 is, as in Fig. 7, moves on a circular segment-shaped vehicle trajectory K, the fourth sensor coordinate system S4, which moves in the direction of the vehicle's longitudinal center axis and thus away from the center of curvature of the vehicle trajectory K, experiences a higher radial velocity than the originally calibrated fourth sensor coordinate system S4' would experience.

[0065] Consequently, a fourth object velocity X4 is also determined by applying the extrinsically calibrated homogeneous coordinate transformation. Although this velocity is not changed in direction, it is changed in magnitude (in this example: increased) compared to the fourth object velocity X4' in the calibrated pose.

[0066] This difference can be determined by comparing the determined fourth object speed X4 with the remaining object speeds X1 to X3, X5 to X7 (which correspond in magnitude to the fourth object speed X4' in the calibrated pose) or with at least one statistical measure derived from the total of all object speeds X1 to X7, as shown by Fig. 5 has already been explained.

[0067] Thus, even a mere offset of a sensor coordinate system S4 relative to the pose in which the extrinsic calibration was performed can be detected using the proposed method. One advantage of this method is that an extrinsic calibration can be reliably validated without requiring the measurement of an identical reference object from multiple sensor coordinate systems S1 to S7. This can improve the reliability of determining a vehicle pose using independent sensors. List of reference symbols 1 vehicle BSS motion estimation step C0 decalibrated state C1 calibrated state D1 to Dn first to n-th sensor data E Decision step K Vehicle trajectory P ext extrinsic parameters PS parameterization step S1 to S7 first to seventh sensor coordinate systems S4' calibrated fourth sensor coordinate system Sn n-th sensor coordinate system TS transformation step V Vehicle coordinate system V1 to V7 first to seventh relative velocity Vn n-th relative velocity X1 to X7 object speed X4' calibrated fourth object speed Xn n-th object speed

Claims

[1] Method for validating a plurality of environment detection sensors rigidly connected to a vehicle (1), which are designed to detect a relative speed (V1 to V7, Vn) of at least one object in the environment of the vehicle (1), each related to a sensor coordinate system (S1 to S7, Sn), and - in an extrinsic calibration, a homogeneous coordinate transformation is determined for each sensor coordinate system (S1 to S7, Sn) to convert coordinates of the sensor coordinate system (S1 to S7, Sn) into coordinates of a vehicle coordinate system (V) rigidly connected to the vehicle (1), - using homogeneous coordinate transformations ◯ for each relative speed (V1 to V7, Vn) an object speed (X1 to X7, Xn) related to the vehicle coordinate system (V) is determined and / or ◯ at least one parameter of a motion model of the vehicle (1) is determined from the plurality of relative speeds (V1 to V7, Vn), characterized by , that - a decalibrated state (C0) is assigned to the plurality of environment detection sensors when the object speeds (X1 to X7, Xn) deviate from each other and / or from the movement model of the vehicle (1) by more than a predetermined amount and - Deviations of the object speeds (X1 to X7, Xn) are determined as a pairwise vector difference and the maximum pairwise vector difference in terms of magnitude and / or direction is compared with a predetermined magnitude difference or a predetermined angular difference. [2] Method according to one of the preceding claims, characterized by , that - a vehicle speed related to the vehicle coordinate system (V) is determined as a movement model of the vehicle (1) in terms of magnitude and direction, - a target relative speed is determined from the vehicle speed according to the respective assigned homogeneous coordinate transformation for each sensor coordinate system (S1 to S7, Sn) and - at least one target relative speed is compared with the relative speed (V1 to V7, Vn) recorded for the respective sensor coordinate system (S1 to S7, Sn) in terms of magnitude and / or direction. [3] Method according to one of the preceding claims, characterized by that a trustworthiness of the assignment of the decalibrated state (C0) and / or a trustworthiness of the assignment of a complementary calibrated state (C1) is statistically determined from the majority of the relative velocities (V1 to V7, Vn). [4] Vehicle (1) comprising at least one computing unit and a plurality of environment detection sensors, which are designed to detect a relative speed (V1 to V7, Vn) of at least one object detected in the environment of the vehicle (1) related to a sensor coordinate system (S1 to S7, Sn) of the respective environment detection sensor, characterized by that the environmental detection sensors and the at least one computing unit are configured to carry out the method according to one of the preceding claims. [5] Vehicle (1) according to claim 4, characterized by that the at least one computing unit is designed as a control unit.

Images