METHOD AND DEVICE FOR EVALUATING THE PERFORMANCE OF AT LEAST ONE ENVIRONMENTAL SENSOR OF A VEHICLE AND VEHICLE WITH SUCH A DEVICE

Description

[0001] The invention relates to a method for evaluating a performance of at least one environmental sensor of a vehicle according to the features of the preamble of claim 1, a device for evaluating a performance of at least one environmental sensor of a vehicle and a vehicle.

[0002] From the prior art, as described in DE 10 2017 006 260 A1, a method for determining detection properties of at least one environmental sensor in a vehicle and a vehicle equipped to carry out such a method are known.The method comprises reading a plurality of environmental sensors of a vehicle in each case in a plurality of measuring cycles, identifying an identical target object in each measuring cycle of the plurality of measuring cycles, wherein for at least one environmental sensor to be tested of the plurality of environmental sensors, it is checked for each measuring cycle of the plurality of measuring cycles whether the target object was detected by the environmental sensor to be tested in the respective measuring cycle, forming a detection number for the environmental sensor to be tested as the number of those measuring cycles of the plurality of measuring cycles in which the target object was detected by the environmental sensor to be tested, and determining a functional measure for the environmental sensor to be tested from the detection number.

[0003] It is known from US 2015 / 347 096 A1 that a false negative rate of an environmental sensor of a vehicle is a sensor property provided by a manufacturer of the environmental sensor.

[0004] It is known from US 2021 / 117 787 A1 that a false negative rate of a sensor, in particular a lidar sensor of a vehicle, can be determined based on the proportion of objects not detected by the sensor out of the total number of objects actually present.

[0005] From US 2021 / 101 624 A1, a method for creating an occupancy map from sensor data of several sensors of an autonomous vehicle is known, wherein it is provided that threshold values ​​for false negative rates are determined and the false negative rates are determined based on detected objects and the deviation from ground truth data.

[0006] The invention is based on the object of specifying a method for evaluating a performance of at least one environmental sensor of a vehicle that is improved compared to the prior art, a device for evaluating a performance of at least one environmental sensor of a vehicle that is improved compared to the prior art, and a vehicle that is improved compared to the prior art.

[0007] The object is achieved according to the invention by a method for evaluating a performance of at least one environmental sensor of a vehicle having the features of claim 1, a device for evaluating a performance of at least one environmental sensor of a vehicle having the features of claim 8 and a vehicle having the features of claim 9.

[0008] Advantageous embodiments of the invention are the subject of the subclaims.

[0009] In a method according to the invention for evaluating the performance of at least one environmental sensor or several environmental sensors of a vehicle, in particular of a vehicle equipped for automated driving, in particular for highly automated or autonomous driving, the following method steps are carried out: Detecting, in particular continuously detecting, false negatives of the environmental sensor with respect to at least one reference object, classifying the false negatives detected during a predetermined evaluation period according to their temporal length and assigning each to one of several predetermined time classes, determining a false negative rate of the respective time class for each of the time classes, which indicates how often false negatives of the respective time class occur during the predetermined evaluation period, specifying a false negative threshold for each of the time classes, determining for each time class how often the false negative rate of the respective time class exceeds the false negative threshold specified for this time class, assessing the performance of the environmental sensor as impaired if, in particular within a predetermined period, for example one hour,the number of exceedances of the false negative threshold specified for the respective time class by the false negative rate is higher than a value specified for the respective time class, in particular a frequency threshold specified for the respective time class.

[0010] A device according to the invention for evaluating the performance of at least one environmental sensor of a vehicle, in particular of a vehicle configured for automated driving, in particular for highly automated or autonomous driving, is designed and configured to carry out the method. In particular, the device comprises a processing unit which is designed and configured to carry out the method. The device also comprises, in particular, the at least one environmental sensor of the vehicle, in particular a plurality of environmental sensors of the vehicle. The respective environmental sensor is designed, for example, as a radar sensor, lidar sensor, or camera sensor, wherein the device can comprise a plurality of identically or differently designed environmental sensors.

[0011] A vehicle according to the invention comprises this device. The vehicle is particularly designed and configured to perform automated driving, in particular highly automated or autonomous driving.

[0012] The at least one or the respective environmental sensor is, in particular, an environmental detection sensor, in particular for detecting the external environment of the vehicle. The performance of the at least one environmental sensor is also referred to as sensor performance.

[0013] Reference objects are selected objects suitable for evaluating performance with regard to false negatives. The reference object can be selected according to the following criteria: The object exists, i.e. it is detected by at least one of the environmental sensors, the object should also be detected by the environmental sensor to be evaluated, i.e. it lies within its detection range, and the object is not hidden from the environmental sensor to be evaluated.

[0014] The reference objects are therefore for environmental sensors that have a detection range oriented straight or diagonally forward, in particular objects that are located directly in front of the vehicle and to the side of the vehicle. Examples of objects that are not suitable as reference objects include an object that is hidden from the respective environmental sensor and an object that lies outside the detection range, i.e., the detection range, of the environmental sensor. Such objects are not suitable for checking the detection performance, i.e., the detection capability, of an environmental sensor because they cannot be detected by the environmental sensor and are therefore not detected by the environmental sensor.

[0015] The evaluation of the performance of the at least one environmental sensor is therefore based in the solution according to the invention on a, in particular continuous, determination of false negatives of the environmental sensor and on a classification of the durations of the determined false negatives, i.e. non-detection durations, according to predetermined time classes. False negative means that an environmental sensor does not detect an object at a location, although an object is actually present at that location. A false negative is therefore a measurement result which indicates that the environmental sensor has not detected an object visible to it in a measurement cycle. For each of the time classes, a false negative rate of the respective time class is determined, which indicates how many times false negatives of the respective time class occur during the predetermined evaluation period. A false negative threshold is defined for each of the time classes.For each time class, the number of times the false negative rate exceeds the false negative threshold defined for that time class is determined. If the determined number of violations exceeds a specified value for that time class, performance is assessed as impaired.

[0016] During automated, particularly highly automated or autonomous, driving operation, the vehicle, in particular a system of the vehicle for carrying out the automated, particularly highly automated or autonomous, driving operation, must detect whether its environmental sensors meet specified requirements regarding their performance with regard to object detection, for example, specified detection rates. This is made possible by the solution according to the invention. If these requirements are not met, the automated, particularly highly automated or autonomous, driving operation is adapted to the reduced performance, for example by reducing a maximum permissible driving speed, or the automated, particularly highly automated or autonomous, driving operation is terminated and vehicle control is transferred to a driver.

[0017] The solution according to the invention thus safeguards automated, in particular highly automated or autonomous, driving operation by only permitting it if the performance of the at least one or the respective environmental sensor, i.e. the sensor performance, meets specified requirements.

[0018] In one possible embodiment, the method steps are performed for each reference object using multiple reference objects and / or for each environmental sensor using multiple environmental sensors. Thus, using the described solution, all of the vehicle's environmental sensors can be evaluated with regard to their performance. Using all reference objects improves the performance evaluation.

[0019] In one possible embodiment, detection results from other environmental sensors of the vehicle regarding the reference object are taken into account to detect false negatives. This makes it possible, in particular, to determine whether a given object is a suitable reference object and to determine whether the object should be detectable by the environmental sensor being evaluated, so that the false negatives for the environmental sensor being evaluated can be correctly determined.

[0020] In one possible embodiment, the detection results of a sensor data fusion from the vehicle's other environmental sensors are used. This increases the certainty in determining false negatives for the environmental sensor being evaluated and thus improves the assessment of its performance.

[0021] In one possible embodiment, the performance of an environmental sensor of the vehicle, designed as a radar sensor, lidar sensor or camera sensor, is evaluated.

[0022] In one possible embodiment, the performance of an environmental sensor of the vehicle is evaluated, which has a detection range oriented straight or diagonally forward or straight or diagonally backward. This allows other road users who are located straight or diagonally in front of the vehicle or straight or diagonally behind the vehicle for an extended period of time, in particular other road users driving ahead or following behind, to be used as reference objects.

[0023] In one possible embodiment, if the performance is assessed as impaired, the automated, in particular highly automated or autonomous, driving operation is adapted to the impaired performance, in particular, the vehicle's speed is reduced or the automated driving operation is terminated. This ensures that the automated, in particular highly automated or autonomous, driving operation is carried out only with sufficiently powerful environmental sensors and thus with a high level of safety.

[0024] The described solution is based on the idea that a false negative rate exceeding a certain false negative threshold can be more easily tolerated with a short non-detection time than with a long non-detection time. The performance of an environmental sensor is therefore advantageously evaluated depending on the false negative rate of the various time classes.

[0025] The requirements that a specific environmental sensor must fulfill are therefore advantageously specified as the maximum number of times within a given period, i.e., how frequently, the false negative rate of a specific time class may be equal to a false negative threshold defined for that time class. The false negative threshold is advantageously specified individually for each sensor type, i.e., for the radar sensor, the lidar sensor, and the camera sensor.

[0026] If the environmental sensor meets all these requirements, its performance is assessed as unaffected. Otherwise, its performance is assessed as impaired.

[0027] Advantageously, the requirements of the other environmental sensors are specified and tested in an analogous manner.

[0028] Based on the detected false negatives of an environmental sensor, for example, other variables for evaluating the performance can be calculated, alternatively or additionally, for example an average duration between the false negatives of an environmental sensor and / or an average duration between the false negatives of a certain false negative class, i.e. time class, of an environmental sensor.

[0029] For example, based on the false negatives detected by an environmental sensor in a specific time window, further statistics for longer time windows can be determined.

[0030] The described solution is therefore advantageously based on the following idea: False negatives from a selected environmental sensor are continuously recorded. A false negative is a measurement result indicating that the selected environmental sensor has not detected at least one reference object visible to the environmental sensor during a measurement cycle. Information about the presence of the reference objects visible to the environmental sensor is provided by other environmental sensors.

[0031] The false negatives recorded during a given evaluation period are classified according to their length of time in order to assign them to one of several predefined time classes.

[0032] For each of the specified time classes, the number of false negatives detected within the specified evaluation period is determined. The number of false negatives detected within a specific time class corresponds to the false negative rate of that time class.

[0033] A false negative threshold is defined for each time class.

[0034] A frequency threshold is defined for each time class.

[0035] For each time class, it is determined whether the false negative threshold of the respective time class is exceeded by the false negative rate of the respective time class and with what frequency, in particular how many times per hour, these exceedances occur. If the determined frequency exceeds the frequency threshold defined for the respective time class, the performance of the environmental sensor is assessed as impaired.

[0036] The above steps are advantageously performed for each existing reference object and for each of the environmental sensors to be checked.

[0037] Embodiments of the invention are explained in more detail below with reference to a drawing.

[0038] It shows: Fig. 1 schematically shows a traffic situation with a vehicle and several reference objects designed as other vehicles in an environment of the vehicle.

[0039] Figure 1 shows a schematic representation of a traffic situation with a vehicle F and several reference objects RO1, RO2, RO3 designed as other vehicles in an environment of the vehicle F. The vehicle F is in particular a vehicle F set up for automated driving, in particular for highly automated or autonomous driving.

[0040] The vehicle F comprises a device 2 which is designed and configured to carry out a method described below for evaluating a performance of at least one environmental sensor s1, si, sz or a plurality of environmental sensors s1, si, sz of such a vehicle F. With regard to the environmental sensors s1, si, sz, i is a running variable from 1 to z, i.e. there are s1 to sz environmental sensors on the vehicle F. The majority of the environmental sensors s1, si, sz with i=1 to z present on the vehicle F are therefore also referred to below as environmental sensors s1 to sz. The respective environmental sensor is referred to below as environmental sensor s1 to sz or environmental sensor si if it is not specifically referred to, for example as s1.

[0041] The device 2 comprises, in particular, a processing unit 3, which is coupled, in particular, to the environmental sensors s1 to sz. In particular, this processing unit 3 is designed and configured to carry out this method. The vehicle F, in particular its device 2, also comprises, in particular, the at least one environmental sensor s1 to sz, in particular a plurality of environmental sensors s1 to sz. The respective environmental sensor s1 to sz is designed, for example, as a radar sensor, lidar sensor, or camera sensor, wherein a plurality of identically or differently designed environmental sensors s1 to sz can be provided. The at least one or respective environmental sensor s1 to sz is, in particular, an environmental detection sensor, in particular for detecting an external environment of the vehicle F. The performance of the at least one or respective environmental sensor s1 to sz is also referred to as sensor performance.

[0042] The procedure involves the following steps in particular: Detecting, in particular continuously detecting, false negatives of the environmental sensor s1 to sz with respect to at least one reference object RO1, RO2, RO3, classifying the false negatives detected during a predetermined evaluation period according to their temporal length and assigning each to one of several predetermined time classes, determining a false negative rate of the respective time class for each of the time classes, which indicates how often false negatives of the respective time class occur during the predetermined evaluation period, specifying a false negative threshold for each of the time classes, determining for each time class how often the false negative rate of the respective time class exceeds the false negative threshold specified for this time class, assessing the performance of the environmental sensor s1 to sz as impaired if, in particular within a predetermined period, for example one hour,the number of exceedances of the false negative threshold specified for the respective time class by the false negative rate is higher than a value specified for the respective time class, in particular a frequency threshold specified for the respective time class.

[0043] Particularly in highly automated driving mode, the vehicle F or a system of the vehicle F must detect whether its environmental sensors s1 to sz meet specified requirements regarding their performance with regard to object recognition, e.g., detection rates, in order to carry out this highly automated driving mode. If these requirements are not met, i.e., if performance is assessed as impaired, the highly automated driving mode is adapted to the reduced performance, e.g., by reducing the maximum permissible driving speed, or the highly automated driving mode is terminated and vehicle control is transferred to a driver. The method described here enables the evaluation of the performance of the respective environmental sensors s1 to sz.

[0044] Methods for determining the detection rates of environmental sensors s1 to sz are already known from the prior art, for example from DE 10 2017 006 260 A1. In this prior art, the determination is based on a statistical evaluation of object detections by the various environmental sensors s1 to sz.

[0045] When evaluating sensor detections, a distinction is usually made between false positives, false negatives, and true positives. False positive means that an environmental sensor s1 to sz detects an object at a location even though there is actually no object there. This is also referred to as a ghost object. False negative means that an environmental sensor s1 to sz does not detect an object at a location even though there is actually an object there. True positive means that an environmental sensor s1 to sz detects an object at a location that is actually present at that location. True positives are referred to as detection rates in the above-mentioned prior art.

[0046] The evaluation of the performance of an environmental sensor s1 to sz is based on the continuous detection of false negatives. A false negative is a measurement result indicating that the environmental sensor s1 to sz failed to detect an object visible to it during a measurement cycle—in the example shown, the respective reference object RO1, RO2, RO3.

[0047] Furthermore, non-detection durations, i.e. the durations of false negatives, are determined. The non-detection durations are classified according to their duration and each assigned to one of several time classes. The number of false negatives that occur in each time class during a specified evaluation period is determined. The number of false negatives detected in a specific time class relative to the evaluation period is referred to as the false negative rate (FN rate) of the respective time class. A false negative threshold (FN threshold) is defined for each time class. For each time class, the number of times the false negative rate of the respective time class exceeds the false negative threshold defined for this time class is determined. If the determined number of exceedances is higher than a specified value for the respective time class, the performance is assessed as impaired.

[0048] Advantageously, several environmental sensors s1 to sz are used for object detection, and objects detected in predetermined spatial areas of the vehicle's surroundings are identified as reference objects RO1, RO2, RO3, i.e. as relevant objects. The objects are detected by sensor data fusion. Thus, to evaluate the performance of a respective environmental sensor s1 to sz, detection results from several or all other environmental sensors s1 to sz of the vehicle F are advantageously also used, in particular to detect false negatives with regard to the respective reference object RO1, RO2, RO3. These detection results from the other environmental sensors s1 to sz are advantageously subjected to sensor data fusion, and the fused sensor data is then used for the described purpose.

[0049] The specified spatial areas are regions of interest, i.e. regions of interest that are located in the surroundings of the vehicle F, for example immediately ahead, to the front left, to the front right, within the sensor range. In these spatial areas there are objects that are relevant to the vehicle F, i.e. to which the vehicle F must react quickly or to which the system must react with a specific target behavior. The method described here is therefore used in particular to evaluate the performance of one or more environmental sensors s1 to sz of the vehicle F, which have a detection range that is aligned straight or diagonally forward.

[0050] Reference objects RO1, RO2, and RO3 are selected objects suitable for conducting a specific performance test, for example, assessing performance with regard to false negatives. Reference objects RO1, RO2, and RO3 can be selected according to the following criteria: The object exists, ie it is detected by at least one of the environmental sensors s1 to sz, the object should also be detected by the environmental sensor s1 to sz to be evaluated, i.e. it lies in its detection range, and the object is not hidden for the environmental sensor s1 to sz to be evaluated.

[0051] The reference objects RO1, RO2, RO3 are therefore, in particular, objects that are located directly in front of the vehicle F and to the side of the vehicle F. Examples of objects that are not suitable as reference objects RO1, RO2, RO3 are, for example, an object that is obscured for the respective environmental sensor s1 to sz and an object that lies outside the detection range of the environmental sensor s1 to sz. Such objects are not suitable for checking the detection performance, i.e., the detection capability, of an environmental sensor s1 to sz, since they cannot be detected by the environmental sensor s1 to sz and are therefore not detected by the environmental sensor s1 to sz.

[0052] In the example according to Figure 1 the reference objects RO1, RO2, RO3 are other vehicles located directly and laterally in front of vehicle F.

[0053] The multiple environmental sensors s1 to sz are, for example, a radar sensor, a lidar sensor, or a camera sensor. Combinations of multiple identical or different environmental sensors s1 to sz are possible.

[0054] For each of the environmental sensors s1 to sz, the false negative rate (FN rate) is determined based on the false negatives detected by the respective environmental sensor s1 to sz. The determination is carried out over a specified time window, i.e., over a specified evaluation period. This specified time window, i.e., the specified evaluation period, comprises a specified number of measurement cycles. The determination is carried out according to the following equation: FN si = N FN si m j

[0055] Here, FN si represents the FN rate. N(FN) si represents the number of times the environmental sensor si failed to detect a reference object RO1, RO2, RO3, i.e., the number of detected non-detections. mj< denotes the number of measurement cycles performed during the evaluation period. This number is a measure of the duration of the evaluation period.

[0056] Furthermore, the duration of the detected non-detections is determined for the environmental sensor si, i.e., a non-detection duration L si is determined. The non-detection duration L si corresponds to the number of consecutive measurement cycles in which a reference object RO1, RO2, RO3 was not detected by the environmental sensor si.

[0057] The non-detection periods L si are classified according to their temporal length. For this purpose, a series of time classes Lk, k = 1, ... n are defined, and the determined non-detection periods are assigned to one of these time classes Lk according to their temporal length.

[0058] For each time class Lk, the number of false negatives of the respective time class Lk occurring within the evaluation period is determined, and based on this, the false negative rate of the respective time class is calculated as follows: FN Lk si = N Lk T

[0059] Here, FN(Lk) si represents the false negative rate of time class Lk. The index si indicates that this false negative rate of time class Lk was determined for the ambient sensor si. N(Lk) represents the number of false negatives assigned to time class Lk during the evaluation period. T represents the duration of the evaluation period or, equivalently, the number of measurement cycles during the evaluation period. The false negative rate of time class Lk thus corresponds to the result of classifying the false negative rate according to the time classes.

[0060] Another possible classification of the false negative rate can be based on object types, for example false negative rate of time class Lk regarding cars, false negative rate of time class Lk regarding trucks, and / or on the basis of other object properties, for example object speed.

[0061] The following Table 1 illustrates how the performance of an environmental sensor s1 is evaluated for an environmental sensor s1 and a reference object RO1, RO2, RO3. The environmental sensor s1 is, for example, a lidar sensor, and the reference object RO1, RO2, RO3 is, for example, the vehicle immediately ahead in Figure 1 , i.e. the reference object RO1.

[0062] In this example, the evaluation period comprises nine measuring cycles. The reference object RO1 could be identified in each of the measuring cycles using the environmental sensors s1 to sz present in vehicle F. The evaluation period has been chosen to be short for illustrative purposes; in a real application, it can be chosen to be much longer. The current measuring cycle is designated t0, and the previous eight measuring cycles are designated t-1 to t-8. Table 1 shows the measuring cycles and the detections made in each measuring cycle. 1 indicates that the environmental sensor s1 detected the reference object RO1 in the respective measuring cycle. 0 indicates that the environmental sensor s1 did not detect the reference object RO1 in the respective measuring cycle.

[0063] For example, three time classes L1, L2 and L3 are defined as follows: Non-detection times with 0 <L s1 <2 werden der Zeitklasse L1 zugeordnet, Nicht-Detektionsdauern mit L s1 =2 werden der Zeitklasse L2 zugeordnet, Nicht-Detektionsdauern mit L s1 > 2 are assigned to time class L3.

[0064] The time classes L1, L2, L3 are also referred to as FN classes below. Table 1: measuring cycle t -8 t -7 t -6 t -5 t -4 t -3 t -2 t -1 t 0 Detection 1 0 1 1 1 1 1 1 1

[0065] For this example, FN s 1 = 1 9 Zyklen and L s 1 = 1

[0066] This means that the reference object RO1 was not detected by the ambient sensor s1 in one of the nine observed measurement cycles. It was not detected in a maximum of one measurement cycle, here in measurement cycle t -7 .

[0067] The determined non-detection duration L s1 = 1 is assigned to the time class L1 according to the classification defined above.

[0068] Thus, a false negative of time class L1 is detected in one out of nine measurement cycles.

[0069] The result for the false negative rate of time class L1 is FN L 1 s 1 = 1 9 Zyklen

[0070] No false negatives are detected in time classes L2 or L3. The false negative rates for time classes L2 and L3 are therefore: FN L 2 s 1 = 0 / 9 Zyklen FN L 3 s 1 = 0 / 9 Zyklen

[0071] The following Table 2 shows another example of a measurement: Table 2: measuring cycle t -8 t -7 t -6 t -5 t -4 t -3 t -2 t -1 t 0 Detection 1 0 0 1 1 1 1 1 1

[0072] In this example, the false negative rate is FN s 1 = 2 9 Zyklen but a longer non-detection period L s 1 = 2

[0073] This means that the reference object RO1 was not detected by the environmental sensor s1 in two of nine observed measurement cycles. It was not detected in a maximum of two consecutive measurement cycles.

[0074] The determined non-detection duration L s1 =2 is assigned to the time class L2 according to the classification defined above.

[0075] There is only one false negative of time class L2, determined in two of nine measurement cycles.

[0076] The false negative rate of time class L2 is therefore: FN L 2 s 1 = 1 9 Zyklen

[0077] No false negatives are detected for time classes L1 or L3. The false negative rates for time classes L1 and L3 are therefore: FN L 1 s 1 = 0 / 9 Zyklen FN L 3 s 1 = 0 / 9 Zyklen

[0078] When measured according to the following Table 3 Table 3: measuring cycle t -8 t -7 t -6 t -5 t -4 t -3 t -2 t -1 t 0 Detection 1 0 0 0 1 1 1 1 1 you get the FN rate FNs1=39 cycles but an even longer non-detection period Ls1=3

[0079] The determined non-detection duration L s1 =3 is assigned to the time class L3 according to the classification defined above.

[0080] There is a false negative of time class L3.

[0081] The false negative rate of time class L3 is therefore: FN L 3 s 1 = 1 9 Zyklen

[0082] No false negatives are detected for time classes L1 or L2. The false negative rates for time classes L1 and L2 are therefore: FN L 1 s 1 = 0 / 9 Zyklen FN L 2 s 1 = 0 / 9 Zyklen

[0083] When measuring according to the Table 4: measuring cycle t -8 t -7 t -6 t -5 t -4 t -3 t -2 t -1 t 0 Detection 1 0 0 0 1 1 0 1 0 you get the false negative rate FNs1=59 cycles and you get two non-detection periods Ls1=3 and Ls1=1

[0084] The determined non-detection duration L s1 =3 is assigned to the time class L3 and the non-detection duration L s1 =1 is assigned to the time class L1.

[0085] There is one false negative of time class L3 and two false negatives of time class L1.

[0086] The false negative rates of time classes L1 and L3 are therefore: FN L 1 s 1 = 2 9 Zyklen FN L 3 s 1 = 1 9 Zyklen

[0087] These measurements are performed for all environmental sensors s1 to sz.

[0088] The described solution is based on the idea that a false negative rate exceeding a certain false negative threshold can be more easily tolerated with a short non-detection time than with a long non-detection time. The performance of an environmental sensor s1 to sz is therefore evaluated depending on the FN rates of the different time classes.

[0089] The requirements that a specific environmental sensor s1 to sz must fulfill are the maximum number of times within a given period, i.e., the maximum frequency, that the false negative rate of a specific time class Lk may be equal to a false negative threshold defined for this time class Lk. The false negative threshold is specified individually for each sensor type.

[0090] For example, the requirement for an environmental sensor is s1 to sn, that the false negative rate of time class L1 may reach the false negative threshold of time class L1 a maximum of five times per hour and may not be higher than this false negative threshold, that the false negative rate of time class L2 may reach the false negative threshold of time class L2 a maximum of three times per hour and may not be higher than this false negative threshold, and that the false negative rate of time class L3 may reach the false negative threshold of time class L3 a maximum of once per hour and may not be higher than this false negative threshold.

[0091] If the environmental sensor s1 to sz meets all these requirements, its performance is assessed as not impaired. Otherwise, the

[0092] Performance is assessed as impaired. Performance is thus assessed as impaired if it is determined that the false negative rate of time class L1 exceeds the false negative threshold defined for this time class L1 more than five times per hour, or if the false negative rate of time class L2 exceeds the false negative threshold defined for this time class L2 more than three times per hour, or if the false negative rate of time class L3 exceeds the false negative threshold defined for this time class L3 more than once per hour.

[0093] The requirements of the other environmental sensors s1 to sz are specified and tested in an analogous manner.

[0094] Based on the detected false negatives of an environmental sensor s1 to sz, further variables can be calculated to evaluate the performance, for example the average duration between the false negatives of an environmental sensor s1 to sz or the average duration between the false negatives of a specific false negative class (time class) of an environmental sensor s1 to sz.

[0095] Based on the detected false negatives of an environmental sensor s1 to sz in a specific time window, further statistics for longer time windows can be determined.

[0096] The described solution is based on the following idea: False negatives from a selected environmental sensor si are continuously recorded. A false negative is a measurement result indicating that the selected environmental sensor si has not detected at least one reference object RO1, RO2, or RO3 visible to the environmental sensor si during a measurement cycle. Information about the presence of the reference objects RO1, RO2, or RO3 visible to the environmental sensor si is provided by other environmental sensors s1 to sz.

[0097] The false negatives recorded during a given evaluation period are classified according to their time length in order to assign them to one of several given time classes Lk with k = 1, ... n.

[0098] For each of the specified time classes Lk with k = 1, ... n, the number of times false negatives of the respective time class are detected within the specified evaluation period is determined. The number of false negatives detected within the evaluation period of a specific time class corresponds to the false negative rate of that time class.

[0099] For each time class Lk with k = 1, ... n a false negative threshold is defined.

[0100] For each time class Lk with k = 1, ... n a frequency threshold is defined.

[0101] For each time class Lk with k = 1, ... n, it is determined whether the false negative threshold of the respective time class is exceeded by the false negative rate of the respective time class Lk and with what frequency, i.e., how many times per hour these exceedances occur. If the determined frequency exceeds the frequency threshold defined for the respective time class, the performance of the environmental sensor si is assessed as impaired.

[0102] The above steps are performed for each existing reference object RO1, RO2, RO3 and for each of the environmental sensors s1 to sz to be checked.

Claims

1. Method for evaluating a performance of at least one environmental sensor (s1 to sz) of a vehicle (F), in particular of a vehicle (F) set up for an automated driving operation, in particular for a highly automated driving operation, characterized by the method steps of - detecting false negatives of the environmental sensor (s1 to sz) with respect to at least one reference object (RO1, RO2, RO3), - classifying the false negatives recorded during a given evaluation time period according to their length of time and assigning each to one of a plurality of given time classes, - determining a false negative rate of the particular time class for each of the time classes, which indicates how often false negatives of the particular time class occur during the given evaluation time period, - presetting a false negative threshold for each of the time classes, - determining for each time class how often the false negative rate of the particular time class exceeds the false negative threshold given for this time class, - evaluating the performance of the environmental sensor (s1 to sz) as impaired if the determined number of times the false negative threshold given for the particular time class is exceeded by the false negative rate is higher than a value given for the particular time class.

2. Method according to claim 1, characterized in that the method steps are carried out for each reference object (RO1, RO2, RO3) in the case of a plurality of reference objects (RO1, RO2, RO3) and / or are carried out for each environmental sensor (s1 to sz) in the case of a plurality of environmental sensors (s1 to sz).

3. Method according to any of the preceding claims, characterized in that to detect false negatives, detection results of other environmental sensors (s1 to sz) of the vehicle (F) with respect to the reference object (RO1, RO2, RO3) are taken into account.

4. Method according to claim 3, characterized in that the detection results of a sensor data fusion of the other environmental sensors (s1 to sz) of the vehicle (F) are used.

5. Method according to any of the preceding claims, characterized in that the performance of an environmental sensor (s1 to sz) of the vehicle (F) designed as a radar sensor, lidar sensor or camera sensor is evaluated.

6. Method according to any of the preceding claims, characterized in that the performance of an environmental sensor (s1 to sz) of the vehicle (F) is evaluated which has a detection range oriented straight or diagonally forward or straight or diagonally backward.

7. Method according to any of the preceding claims, characterized in that the automated driving operation is adapted to the impaired performance or terminated if the performance is evaluated as impaired.

8. Device (2) designed and set up for carrying out a method according to any of the preceding claims.

9. Vehicle (F) having a device (2) according to claim 8.

10. Vehicle (F) according to claim 9, designed and set up for carrying out an automated driving operation.