Method of monitoring a vehicle system for identifying a vehicle environment

Abstract

The invention relates to a method for monitoring a vehicle system for detecting a vehicle environment, the vehicle system having a sensor system with at least two sensor units for detecting the vehicle environment and an evaluation unit for recognizing objects in the vehicle environment by fusing sensor data of the at least two sensor units. The method comprises the following steps: determining a presence probability and a detection probability for each recognized object on the basis of the sensor data, wherein the presence probability indicates at what probability the recognized object represents a real object in the vehicle environment and the detection probability indicates at what probability a recognized object can be detected by the sensor system, wherein the presence probability and the detection probability are determined individually for each of the at least two sensor units; and determining whether the vehicle system is in a robust state on the basis of the presence probability and the detection probability.

Description

Technical Field

[0001] This invention relates to a method for monitoring a vehicle system used to identify the vehicle environment. Furthermore, the invention also relates to an evaluation unit, a computer program, and a computer-readable medium for performing the method, as well as a corresponding vehicle system. Background Technology

[0002] To identify the vehicle's environment, appropriate algorithms can be used to combine sensor data from different vehicle sensors into a common representation of that environment; this is also known as sensor data fusion. The goal of this sensor data fusion is to merge individual sensor data, allowing the strengths of each sensor to be beneficially combined or reducing their individual weaknesses. The ability to accurately identify the environment under different external conditions depends particularly on the robustness of this vehicle system. Summary of the Invention

[0003] In this context, methods, evaluation units, computer programs, and computer-readable media according to the solutions presented herein are proposed. Advantageous extensions and improvements of the solutions presented herein are derived from the specification and described in the dependent claims.

[0004] Advantages of the present invention

[0005] Embodiments of the present invention advantageously enable the robustness of a vehicle system with multiple sensors for identifying the environment to be estimated based on sensor-specific presence and detection probabilities. This avoids erroneous positive results, erroneous negative results, or other incorrect results when identifying objects.

[0006] A first aspect of the invention relates to a method for monitoring a vehicle system for identifying a vehicle environment, wherein the vehicle system has a sensor system and an evaluation unit, the sensor system having at least two sensor units for detecting the vehicle environment, and the evaluation unit for identifying objects in the vehicle environment by fusing sensor data from the at least two sensor units. The method includes the steps of: determining a probability of presence and a probability of detection for each identified object based on the sensor data, wherein the probability of presence indicates the probability that the identified object represents a real object in the vehicle environment, and the probability of detection indicates the probability that the identified object can be detected by the sensor system, wherein the probability of presence and the probability of detection are determined individually for each of the at least two sensor units; and determining whether the vehicle system is robust based on the probability of presence and the probability of detection.

[0007] A vehicle can generally be understood as a partially or fully automated moving machine. For example, a vehicle can be a passenger car, truck, bus, motorcycle, robot, etc.

[0008] The vehicle system can be designed to control the vehicle in a partially or fully automated manner. To this end, the vehicle system can manipulate the vehicle's corresponding actuator systems, such as steering or braking actuators or engine control devices.

[0009] The sensor unit can be, for example, a radar sensor, a lidar sensor, an ultrasonic sensor, or a camera. The sensor system can include sensor units of the same type (e.g., redundant) or different types (e.g., complementary). For example, a combination of a radar sensor and a camera, or a combination of multiple radar sensors with different detection directions, is conceivable.

[0010] The sensor data may include features identified by the respective sensor units, such as the position, velocity, acceleration, extension, or object category of an object in the vehicle environment. These features can be extracted from the raw data of the respective sensor units.

[0011] The sensor data can be fused together to identify the vehicle's environment. Such sensor data fusion can be understood as the following process: using information from different sensor instances to detect and classify objects in the vehicle environment (also known as object differentiation) and estimate the respective states of the objects, that is, predicting the respective states of the objects with a certain probability (also known as trajectory estimation).

[0012] In this context, identified objects can be understood as models of real objects located within the vehicle environment, such as observed vehicles, road markings, pedestrians, etc. Identified objects, along with their respective features, can be stored in an environmental model representing the vehicle environment. This environmental model can be continuously updated based on the sensor data by comparing the predicted states of the objects with current measurements.

[0013] The existence probability can, for example, indicate the probability that a real object produced the measurement history and motion pattern of the identified object.

[0014] The detection probability may depend, for example, on the position of the object relative to the mounting location of the respective sensor unit on the vehicle, or on environmental conditions that may limit the perception of the respective sensor unit.

[0015] By determining the presence probability and the detection probability separately for each sensor unit or sensor instance, the impact of each sensor unit or sensor instance on the overall result of sensor data fusion can be estimated.

[0016] A second aspect of the invention relates to an evaluation unit configured to perform the methods described above and below. Features of the methods may also be features of the evaluation unit, and vice versa.

[0017] A third aspect of the invention relates to a vehicle system configured to perform the methods described above and below. Features of the methods may also be features of the vehicle system, and vice versa.

[0018] Other aspects of the invention relate to a computer program that, when executed by a computer such as the evaluation unit described above, performs the methods described above and below, and to a computer-readable medium on which such a computer program is stored.

[0019] The computer-readable medium can be a volatile or non-volatile data storage device. For example, the computer-readable medium can be a hard disk, a USB storage device, RAM, ROM, EPROM, or flash memory. The computer-readable medium can also be a data communication network that enables the download of program code, such as the Internet or a cloud.

[0020] The features described above and below can also be features of the computer program and / or the computer-readable medium, and vice versa.

[0021] The ideas behind the embodiments of the present invention can be particularly considered to be based on the ideas and findings described below.

[0022] According to one embodiment, the sensor system includes at least a first sensor unit, a second sensor unit, and a third sensor unit. Here, if the probability of presence of an identified object assigned to the first sensor unit is outside the expected range, while the probabilities of presence assigned to the second and third sensor units are both within the expected range, and the detection probabilities assigned to each sensor unit are identified as high, then the vehicle system is determined to be in a non-robust state.

[0023] It is possible that at least the three sensor units are different sensor entities, i.e., sensor types.

[0024] The expected range can be understood as a predefined value or range of values.

[0025] For example, if the detection probability is higher than a detection threshold representing a high detection probability, then the detection probability can be identified as high.

[0026] This avoids false positive or false negative identifications. It also helps identify interference that affects all sensor units, such as undetectable masking.

[0027] According to one implementation, if the detection probability assigned to the first sensor unit and the detection probability assigned to the second sensor unit are identified as low for the identified object, and the detection probability assigned to the third sensor unit and the presence probability assigned to the third sensor unit are identified as high, then the vehicle system is determined to be in a non-robust state.

[0028] For example, if the detection probability is lower than a detection threshold representing a low detection probability, the detection probability can be identified as low.

[0029] For example, if the probability of existence is higher than the existence threshold representing a high probability of existence, then the probability of existence can be identified as high.

[0030] The detection threshold or the presence threshold can vary depending on the sensor unit, for example, depending on the installation location of the sensor unit on the vehicle or the distance between the sensor unit and the object to be identified in the vehicle environment.

[0031] According to one embodiment, the sensor data includes features of objects in the vehicle environment identified by their respective sensor units. A plurality of possible assignment matrices are generated based on the sensor data, each assignment matrix describing a possible assignment between the features and identified objects and / or between the identified objects and at least one preprocessing algorithm. An assignment matrix for updating the identified objects is selected from the plurality of possible assignment matrices. Furthermore, if at least two different assignment matrices and / or two different preprocessing algorithms are alternately selected within a predefined time period, the vehicle system is determined to be in a non-robust state.

[0032] In the assignment matrix, at least one identified object (at least one object hypothesis) can be assigned to each feature (more precisely, each feature hypothesis), and / or at least one preprocessing algorithm can be assigned to each identified object. The assignments can be weighted, for example, by means of cost.

[0033] Updating the environment model based on multiple alternative assignment matrices can also be called a multi-hypothesis scheme.

[0034] Unlike classic multi-hypothesis tracking, here different preprocessing algorithms can be used to process the sensor data, meaning the allocation matrix can select different preprocessing algorithms in certain situations. This allows different variations of sensor data processing to be executed in parallel.

[0035] For example, the preprocessing algorithm can be selected based on the identified object type and / or based on the respective functionality and (depending on the object type and / or environmental conditions) of the preprocessing algorithm.

[0036] Frequent switching between different assignment matrices or preprocessing algorithms (e.g., between preprocessing algorithms that yield the best evaluation) may indicate a decrease in the robustness of environment detection.

[0037] According to one implementation, the identified objects are stored in an environment model representing the vehicle environment. The robustness of the vehicle system is also determined based on the frequency with which the environment model reverts from its current state to a state rated as trustworthy within a predefined time period.

[0038] According to one embodiment, the method further includes: if it has been determined that the vehicle system is in a non-robust state, then placing the vehicle system into a safe mode. The safe mode may include, for example, braking the vehicle, altering the vehicle's planned trajectory, or preventing specific maneuvers of the vehicle, such as changing lanes. Attached Figure Description

[0039] Embodiments of the present invention are described below with reference to the accompanying drawings, which should not be construed as limiting the invention.

[0040] Figure 1 A vehicle having a vehicle system according to an embodiment is shown.

[0041] Figure 2 A flowchart of a method according to an embodiment is shown.

[0042] These figures are schematic only and are not drawn to scale. In these figures, the same reference numerals indicate features that have the same or the same function. Detailed Implementation

[0043] Figure 1 A vehicle 100 with a vehicle system 102 is shown. This vehicle system includes a sensor system and an evaluation unit 110. The sensor system has a first sensor unit 104 (here, a radar sensor), a second sensor unit 106 (here, a lidar sensor), and a third sensor unit 108 (here, a camera) for detecting objects in the environment of the vehicle 100. The evaluation unit 110 is used to evaluate the respective sensor data 112 of the three sensor units 104, 106, and 108. Exemplarily, Figure 1 The sensor system in the middle detects the vehicle 113 traveling in front.

[0044] Additionally, vehicle system 102 may include actuator system 114, such as steering or braking actuators or engine controllers of vehicle 100. Evaluation unit 110 may manipulate actuator system 114 in a suitable manner based on sensor data 112 to, for example, fully automated control of vehicle 100.

[0045] To identify the vehicle 113 traveling ahead as an object, sensor data 112 from different sensor units 104, 106, and 108 are appropriately fused together in the evaluation unit 110. Here, the identified object is stored in an environment model and continuously updated based on the sensor data 112; this is also referred to as tracking. At each time step, the future state of the identified object in the environment model is estimated, and the future state is compared with the current sensor data 112.

[0046] To determine whether the vehicle system 102 is sufficiently robust—that is, capable of correctly identifying objects in the environment of vehicle 100 under different environmental conditions—the evaluation unit 110 determines the presence probability for each identified object based on sensor data 112. This presence probability indicates the probability that the identified object (e.g., a model of the vehicle 113 traveling ahead) matches the real object (here, the actual vehicle 113 traveling ahead). Furthermore, the evaluation unit 110 determines a detection probability for each identified object, indicating the probability that the sensor system can detect the object (here, the vehicle 113 traveling ahead) in the environment of vehicle 100.

[0047] The presence probability and the detection probability are both object-specific and sensor-specific. In other words, the presence probability and the detection probability are determined for each of the three sensor units 104, 106, and 108, so that for each object identified within the scope of sensor data fusion, the respective presence probability and detection probability of the sensor unit or sensor instance participating in the sensor data fusion are known.

[0048] As will be described in more detail below, the presence probability and detection probability are evaluated to assess the robustness of vehicle system 102.

[0049] Figure 2 A flowchart of method 200 is shown, which can be performed by... Figure 1 The vehicle system 102 is executed.

[0050] In the first step 210, as already mentioned, the probability of presence and the probability of detection of the identified object are determined for each of the three sensor units 104, 106, and 108.

[0051] In the second step 220, the existence probability and detection probability of different sensor units are compared with each other to infer the robustness of the vehicle system 102 based on the deviation.

[0052] For example, if the probability of presence assigned to the first sensor unit 104 regarding the identified vehicle 113 traveling ahead is outside the expected range, while the probabilities of presence assigned to the second sensor unit 104 and the third sensor unit 108 are within the expected range, and furthermore, the detection probabilities of all three sensor units 104, 106, and 108 are identified as high, for example because these detection probabilities are all above a predefined threshold, then the vehicle system 102 is identified as not robust. In other words, if the results provided by the first sensor unit 104 deviate from or even contradict the respective results of the second sensor unit 106 and the third sensor unit 108 over a (relatively long) predefined duration, then the vehicle system 102 is identified as lacking robustness.

[0053] Here, the deviation may be due to the fact that the probability of the presence of the first sensor unit 104 is significantly greater than that of the presence of sensor units 106 and 108, which may lead to a false positive result, or that the probability of the presence of the first sensor unit 104 is significantly lower than that of the presence of sensor units 106 and 108, which may lead to a false negative result. Another reason for this deviation may be, for example, unidentified occlusion involving all three sensor units 104, 106, and 108.

[0054] For example, a lack of robustness can also be identified if, for one of sensor units 104, 106, and 108, the probability of the presence of all objects within the field of view is low over a relatively long period of time, but these objects are simultaneously identified by other sensor units. This could indicate a failure of the sensor unit involved.

[0055] Furthermore, a lack of robustness can be identified if a low detection probability is determined for two of the three sensor units 104, 106, and 108 for an identified object, while a high presence probability is simultaneously assigned to the identified object by the third sensor unit. In particular, a lack of robustness can be identified if this occurs multiple times within a predefined time period.

[0056] Additionally, hypotheses derived from multiple hypothesis tracking can be used to identify whether vehicle system 102 is operating robustly.

[0057] One possible indicator of system weakness or system sensitivity is the alternating selection of two or more different (contradictory) hypotheses or models over a relatively short period of time, that is, jumping back and forth between two or more different hypotheses or models.

[0058] Another metric might be that the environment model or one of the objects stored in the environment model regresses from its current state to a rated reliable alternative state for too long or too frequently within a predefined time period.

[0059] The overall confidence level of selected hypotheses about objects identified as real can also be determined at multiple time steps. If the overall confidence level is too low too frequently within a specific time period, a non-robust state of vehicle system 102 can also be inferred.

[0060] As a result of step 220, binary information may be output, which may indicate that the vehicle system 102 is robust or that the vehicle system 102 is not robust.

[0061] Alternatively or additionally, detailed information may also be output, such as possible causes of system weaknesses or system sensitivities, or information about which sensor units or sensor data are involved.

[0062] The detailed information may be used for diagnostic purposes, to reset the sensor units involved, or to test new areas of autonomous vehicles, also known as the Operational Design Domain.

[0063] In response to the identification of reduced or absent robustness of vehicle system 102, vehicle system 102 can be placed into a safe operating mode. This can cause vehicle system 102 to, for example, brake vehicle 100, stop vehicle 100, or prevent certain more complex maneuvers, such as lane changes.

[0064] Finally, it should be noted that terms such as "having" or "comprising" do not exclude other elements or steps, and terms such as "a" or "an" do not exclude multiple. Reference numerals in the claims should not be considered limiting.

Claims

1. A method (200) of monitoring a vehicle system (102) for detecting an environment of a vehicle (100), wherein the vehicle system (102) has a sensor system with at least two sensor units (104, 106, 108) for detecting the environment of the vehicle (100) and an evaluation unit (110) for identifying objects (113) in the environment of the vehicle (100) by fusing sensor data of the at least two sensor units (104, 106, 108), wherein the method (200) comprises: determining (210) a presence probability and a detection probability for each identified object on the basis of the sensor data (112), wherein the presence probability indicates at what probability the identified object represents a real object (113) in the environment of the vehicle (100) and the detection probability indicates at what probability the identified object can be detected by the sensor system, wherein the presence probability and the detection probability are determined individually for each of the at least two sensor units (104, 106, 108); and determining (220) whether the vehicle system (102) is in a robust state on the basis of the presence probability and the detection probability.

2. The method (200) according to claim 1, wherein the sensor system comprises at least a first sensor unit (104), a second sensor unit (106) and a third sensor unit (108); wherein the vehicle system (102) is determined to be in a non-robust state if the presence probability assigned to the first sensor unit (104) with respect to an identified object lies outside an expected range, the presence probability assigned to the second sensor unit (106) and the presence probability assigned to the third sensor unit (108) are both within the expected range and the detection probability assigned to each sensor unit (104, 106, 108) is identified as high.

3. The method (200) according to claim 1, wherein the sensor system comprises at least a first sensor unit (104), a second sensor unit (106) and a third sensor unit (108); wherein the vehicle system (102) is determined to be in a non-robust state if the detection probability assigned to the first sensor unit (104) and the detection probability assigned to the second sensor unit (106) are identified as low, respectively, and the detection probability assigned to the third sensor unit (108) and the presence probability assigned to the third sensor unit (108) are identified as high, respectively.

4. The method (200) according to any one of claims 1 to 3, wherein, the sensor data (112) comprises features of objects (113) in the environment of the vehicle (100) identified by the respective sensor unit (104, 106, 108); wherein a plurality of possible assignment matrices is generated based on the sensor data (112), each assignment matrix describing possible assignments between the features and the recognized objects and / or between the recognized objects and at least one pre-processing algorithm; wherein an assignment matrix is selected from the plurality of possible assignment matrices for updating the recognized objects; wherein it is further determined that the vehicle system (102) is in a non-robust state if at least two different assignment matrices and / or two different pre-processing algorithms are alternately selected within a predefined time period.

5. The method (200) according to any one of claims 1 to 3, wherein the recognized objects are stored in an environment model representing an environment of the vehicle (100); wherein it is further determined whether the vehicle system (102) is in a robust state depending on a frequency with which the environment model falls back from a current state to a state rated as trustworthy within a predefined time period.

6. The method (200) according to any one of claims 1 to 3, further comprising: if it has been determined that the vehicle system (102) is in a non-robust state, putting the vehicle system (102) into a safe mode.

7. An evaluation unit (110) configured to perform the method (200) according to any one of the preceding claims.

8. A vehicle system (102), comprising: a sensor system having at least two sensor units (104, 106, 108) for detecting an environment of a vehicle (100); and the evaluation unit (110) according to claim 7.

9. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method (200) according to any one of claims 1 to 6.

10. A computer readable medium having stored thereon the computer program product according to claim 9.

Images