Method and system for determining the state of the surrounding environment of a vehicle

Abstract

The present invention relates to a method for determining an environment state of a vehicle (10), comprising: receiving sensor data (20) from at least one environment sensor (14) of the vehicle (10); feeding at least one first part (20a) of the sensor data (20) into at least one first classifier (24); generating an intermediate probability (32) from the first part (20a) of the sensor data (20) by the first classifier (24); feeding at least one second part (20b) of the sensor data and the at least one intermediate probability (32) into a second classifier (28); generating a final probability (34a, 34b, 34c) for the environment state from the second part (20b) of the sensor data (20) and the at least one intermediate probability (32) by the second classifier (28).

Description

Technical Field

[0001] The present invention relates to a method and a system for determining the state of the vehicle's surroundings. Background Art

[0002] Driver assistance systems use sensor data from sensors installed in the vehicle in order to monitor the surroundings of the vehicle and thereby obtain information for controlling the vehicle. For this reason, driver assistance systems often use model-based methods in order to be able to achieve good physical comprehensibility and thus simple evaluation from a safety perspective. Driver assistance systems are becoming more and more complex and place higher and higher demands on the available computing performance. Therefore, attempts are being made to replace model-based methods with data-based or learning-based methods. An example of this would be the use of methods based on "deep convolution neural networks".

[0003] Although such "deep learning" methods partially achieve excellent results, it is no longer simple to ensure the desired functionality. Here, the correct estimation of the situation, the correct vehicle characteristics and the functional robustness with respect to sensor noise and unknown objects must be verified and ensured.

[0004] Another challenge associated with methods based on neural networks may be that the first input layer of a neural network is usually not allowed to obtain invalid values. Unfortunately, this condition no longer applies to features calculated from radar data. For example, some characteristics, such as the elevation angle, cannot be reliably calculated for all object angles. Summary of the invention

[0005] The object of the present invention is to improve the output correctness and the robustness of a system for determining a state of a vehicle's surroundings.

[0006] This object is achieved by a method and a system for determining a state of vehicle surroundings and by preferred developments thereof.

[0007] One aspect of the invention relates to a method for determining a state of the surroundings of a vehicle. The method can be automatically implemented by a system of the vehicle, which is, for example, part of a driver assistance system of the vehicle. The system can, for example, execute a corresponding computer program. The vehicle can be a road vehicle, such as a passenger car, a truck, a motorcycle or a bus. The vehicle can be a rail vehicle, such as a tram.

[0008] The state of the surroundings of the vehicle may describe the presence of objects in the surroundings, such as other vehicles, obstacles, roads, etc. and / or conditions such as, for example, a collision hazard. In one case, the state of the surroundings may describe a condition, such as an "obstacle", and the condition may be sensed with one or more probabilities. The state of the surroundings may be described by the presence and / or absence of an object class.

[0009] According to one embodiment of the present invention, the method includes: receiving sensor data from at least one surrounding environment sensor of the vehicle; feeding at least one first part of the sensor data into at least one first classifier; generating an intermediate probability from the first part of the sensor data through the first classifier; feeding at least one second part of the sensor data and at least one intermediate probability into a second classifier; and generating a final probability for the surrounding environment state from the second part of the sensor data and at least one intermediate probability through the second classifier.

[0010] In summary, the method can be implemented by two blocks, wherein the first block includes one or more first classifiers and the second block includes a second classifier. At least a portion of the sensor data is fed into each of the two blocks of the block. The two blocks thus calculate at least one probability for each classifier. In this case, the output from the first block, i.e. the probability, is additionally used as input for the second block. This can increase the correctness and robustness of the entire method.

[0011] In this case, a classifier may be a module and / or a software component which calculates one or more probabilities for surrounding conditions or object classes based on input values, for example relevant parts of sensor data.

[0012] Here, one or more first classifiers from the first block may be specific classifiers that determine a more specific state of the surrounding environment than the second classifier. Therefore, the second classifier from the second block may be regarded as a more general classifier. The probability generated by the classifier from the first block may be referred to as an intermediate probability or a specific probability. The probability generated by the classifier from the second block may be referred to as a final probability or a general probability.

[0013] The first block may include, for example, a plurality of classifiers which generate intermediate probabilities for the presence of a "truck", "passenger car", "motorcycle", "pedestrian", etc. in front of the vehicle. Conversely, the second classifier may output the probability for an "obstacle" in front of the vehicle as the final probability. It should be noted that the method can also be used for the classification of other situations or other safety-critical tasks of a driver assistance system.

[0014] By inputting the probabilities from the first block into the second block in addition to the sensor data in this way, the robustness of the method can be increased, since the more specific results from the first block must be consistent with the sensor data input into the second block.

[0015] Further, the first block and the second block can be executed on different processors, which can improve redundancy or fail-safety. Further, the intermediate probabilities from the first block can be used as inputs to other functions. The classifiers from the first block can be used redundantly in other functions.

[0016] The surrounding environment sensor can be, for example, a radar system, a laser system, an ultrasound system and / or a camera. Therefore, the sensor data can include radar data and / or image data and / or lidar data and / or ultrasound data.

[0017] According to one embodiment of the invention, the method further comprises: feeding at least a portion of the sensor data into a plurality of first classifiers; generating an intermediate probability from a second portion of the sensor data by each of the first classifiers; and feeding the intermediate probability into a second classifier. As already mentioned, the first block can include more than one classifier. The classifiers from the first block can classify specific ambient conditions, which respectively contain information about a general ambient condition, which is classified by the classifiers from the second block.

[0018] According to one embodiment of the present invention, the first classifier and the second classifier determine the corresponding probabilities by means of different classification methods. A plurality of first classifiers may also be based on different classification methods. In general, the classifiers may be different, not only between blocks but also within blocks. Here, a classification method may be understood as an algorithm that analyzes and evaluates input data or sensor data.

[0019] It is possible to train the two classifiers with different training data or to provide the two classifiers with different model data, but the two classifiers work with the same algorithm. In this case, the classifiers work with the same classification method.

[0020] According to one embodiment of the present invention, the first classifier and / or the second classifier is based on a learning-based classification method, which is automatically trained with classified training data. The learning-based classification method can be, for example, a method based on a neural network or based on statistical analysis evaluation. An example of this would be a deep learning method.

[0021] According to one embodiment of the present invention, the first classifier and / or the second classifier is based on a model-based classification method, which includes a model built based on classified training data. In the model-based method, the model can be built manually, that is, by a person.

[0022] For example, the first classifier from the first block (or some of the first classifiers) can be based on a model-based method. The second classifier from the second block can be based on a learning-based method. However, it is also possible that both the first classifier and the second classifier are based on the same method, for example a learning-based method, such as a deep learning method.

[0023] According to one embodiment of the invention, the method further comprises: determining a first feature vector from the sensor data for the first classifier, wherein the first classifier determines the intermediate probability from the first feature vector. Feature vectors can also be determined for a plurality of classifiers from the first block. Feature vectors can be understood as sensor data compressed in a specific manner. Therefore, the method of the classifier does not have to be implemented on all sensor data, but only on feature vectors, which can be significantly shorter or longer than the sensor data used for the classifier.

[0024] According to one embodiment of the present invention, the method further comprises: determining a second feature vector from the sensor data for the second classifier, wherein the second classifier determines the final probability from the second feature vector. The sensor data of the second classifier can also be compressed in this way.

[0025] For example, each classifier (first and / or second classifier) ​​may have an input module which generates a feature vector from the relevant part of the sensor data.

[0026] According to one embodiment of the present invention, the first feature vector and the second feature vector are different. Possibly, the feature vector for the first classifier is generated differently from the feature vector for the second classifier and / or is generated by different sensor data. The use of different feature vectors can improve the robustness and / or efficacy of the method.

[0027] According to one embodiment of the present invention, feature vectors are generated for multiple first classifiers from sensor data respectively. It is also possible that the feature vectors of multiple first classifiers are different. For example, a classifier for "pedestrians" can obtain a feature vector containing additional information about the micro-Doppler characteristics of the radar data (pedestrians usually have high micro-Doppler echoes, which are caused by the movement of the limbs). Classifiers for vehicles, such as "passenger cars", "trucks", etc., can obtain feature vectors that do not contain these micro-Doppler characteristics. In vehicles, such characteristics are weakly present and less important, and therefore classification can be performed without these characteristics.

[0028] According to one embodiment of the invention, a first portion of the sensor data used by at least one first classifier is different from a second portion of the sensor data used by a second classifier.Furthermore, the robustness of the method can be improved by providing the determined sensor data to only one of the blocks.

[0029] According to one embodiment of the present invention, the first part and the second part are mutually independent parts of the sensor data. Possibly, the sensor data is divided and the sensor data provided to the first block or the first classifier is not input into the second classifier and vice versa. In other words, the first part of the sensor data and the second part of the sensor data may be independent data sources. For example, the first part of the sensor data may originate from the first sensor and the second part of the sensor data may originate from the second sensor.

[0030] It is further possible that the first classifier or some of said first classifiers and the second classifier are based on a learning-based method, which is trained on the basis of independent data. This can improve the correctness of the method.

[0031] According to one embodiment of the present invention, the first classifier and / or the second classifier are respectively composed of at least one first subclassifier and one second subclassifier, wherein the probability generated by at least one first subclassifier is fed into the second subclassifier. In other words, one or more of the classifiers may be composed of one or more first classifiers and second classifiers or have a first block and a second block, wherein the output of the first block is fed into the second block, as in the system. The subclassifier may be a classifier that is a component of the upper classifier.

[0032] According to one embodiment of the present invention, at least one first classifier generates more than one intermediate probability. For example, the first classifier may output the probability of "there is a surrounding environment state to be classified" and the probability of "there is no surrounding environment state to be classified".

[0033] According to one embodiment of the present invention, the second classifier generates a first final probability of “whether the surrounding environment state exists.” Usually, the probability can be described as a value between 0 and 1.

[0034] According to one embodiment of the present invention, the second classifier generates a second final probability of “whether the surrounding environment state does not exist.” Generally, the second classifier can generate one, two, three or more final probabilities as output signals.

[0035] According to one embodiment of the present invention, the second classifier generates a third final probability of "whether the sensor data can be classified". Possibly, the third final probability is generated or output together with the first final probability and / or the second final probability. The generation and / or output of the third final probability can be achieved in particular in the following manner: the intermediate probability from the first block is additionally input into the second classifier.

[0036] For example, for an obstacle as the surrounding environment state, the second classifier may output the category “unknown” or the probability of the category in addition to the categories “obstacle” and “non-obstacle”.

[0037] The third final probability increases robustness, since in the case of strongly changing sensor data, the calculated features can deviate strongly from the training data. The same can also apply to objects and / or situations that were not included in the training.

[0038] According to one embodiment of the present invention, the method further comprises: outputting the final probability to a control system of the vehicle and controlling the vehicle based on the final probability. For example, when an obstacle in front of the vehicle has been identified, the vehicle may autonomously start braking.

[0039] Another aspect of the invention relates to a system for determining the surrounding conditions of a vehicle. The system can implement the method as described above and below.

[0040] The system includes: at least one first classifier for receiving at least one first portion of vehicle sensor data and for generating an intermediate probability from the received sensor data; and at least one second classifier for receiving at least one second portion of vehicle sensor data and for generating a final probability about the state of the surrounding environment from the received sensor data and at least one intermediate probability.

[0041] Further, the system may include one or more sensors that generate sensor data.The system may also include a control system, the one or more final probabilities being input into the control system and the control system controlling the vehicle based on the one or more final probabilities.

[0042] It will be appreciated that features of the method as described above and below may also be features of the system and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0044] Figure 1 A vehicle having a classification system according to one specific embodiment of the present invention is schematically shown.

[0045] Figure 2 A classification system according to one embodiment of the present invention is schematically shown.

[0046] Figure 3 A part of a classification system according to one embodiment of the present invention is schematically shown.

[0047] In principle, identical or similar parts are provided with the same reference symbols. DETAILED DESCRIPTION

[0048] Figure 1 A vehicle 10 is shown with a system 12, which includes one or more sensors 14, a driver assistance system 16 and a brake system 18. The one and / or more sensors 14 may include a radar sensor and / or a camera and / or a lidar sensor and / or an ultrasonic sensor. The driver assistance system 16 receives sensor data 20 from the one or more sensors 14, evaluates the sensor data and automatically initiates braking with the aid of the brake system 18 if the sensor data 20 indicates an obstacle in front of the vehicle 10, for example.

[0049] Figure 2 System 12 and in particular components of driver assistance system 16 that are relevant to the method described here are shown in greater detail.

[0050] The system 16 comprises a first block 22 with a plurality of first classifiers 24 and a second block 26 with a second classifier 28. Possibly, the block 22 comprises only one first classifier 24 and the second block 26 comprises a plurality of second classifiers 28. The classifiers 24, 28 can be based on learning-based and / or model-based methods, as further explained above.

[0051] A first portion 20a of the sensor data 20 is fed into a first block 22. A second portion 20b of the sensor data 20 is fed into a second block 26. The portions 20a, 20b of the sensor data 20 may at least partially differ from one another. It is also possible that the portions 20a, 20b are independent of one another, i.e. do not overlap. For example, the portions 20a and 20b may originate from different sensors 14. For increased robustness and consistent classification results, the classifier from the first block 22 and the classifier 28 from the second block 26 may also be trained on partially or completely independent data.

[0052] For each of the classifiers 24, 28, a feature vector 30, 31 is calculated from the corresponding portion 20a, 20b of the sensor data 20. This can be done by the corresponding classifier 24, 28 or by a separate module, which can also generate feature vectors 30, 31 for a plurality of classifiers 24, 28, for example. The feature vectors 30, 31 can encode object properties, which can be derived from the sensor data 20.

[0053] The feature vectors 30, 31 of different classifiers 24, 28 may be different. For example, the feature vector 31 of the classifier 28 may be different from the feature vector 30 of the classifier 24. It is also possible that two feature vectors 30 of two or more classifiers 24 are different from each other. For example, for the classification of pedestrians or cyclists based on radar, micro-Doppler characteristics may be particularly important. On the contrary, for the classification of passenger cars, these characteristics are more likely to be interfering or insignificant. It is therefore possible that one feature vector 30 of a classifier 24 contains micro-Doppler characteristics, while another feature vector does not.

[0054] Each of the classifiers 24 calculates one or more intermediate probabilities 32 from the feature vector 30. Each of the intermediate probabilities 32 may be a value between 0 and 1.

[0055] The block 22 may include a plurality of classifiers 24 as shown here, each performing a fine classification for determining an object type or determining a condition. For example, one of the classifiers 24 may classify data from a corresponding feature vector 30 into the categories "passenger car" or "non-passenger car". Other examples for such classifiers 24 are listed further down in the table.

[0056] The probabilities 32 and the feature vectors 31 of the second block 26 are fed into a classifier 28. The classifier 28 can classify one or more superordinate classes for the subclassification of the classifier 24. The classifier 28 can calculate three final probabilities 34a, 34b, 34c, which respectively indicate the presence of the first superordinate class, the presence of the second superordinate class, and the absence of both superordinate classes.

[0057] For example, the classifier 28 may classify obstacles and non-obstacles and describe final probabilities 34a, 34b, 34c for the categories “obstacle”, “non-obstacle”, and “unknown”.

[0058] The table below shows an example of a classifier 28 and examples of possible intermediate probabilities 32 and final probabilities 34a, 34b, 34c.

[0059]

[0060] Example 1 is an example classification of a passenger car. Here, the passenger car classifier 24 provides a high probability 32 and the other subclassifiers 24 from the first block 22 each provide a low probability 32. Based on this preclassification and the feature vector 31 of the second block 26, the obstacle classifier 28 calculates a high probability 34a for an obstacle. Autonomous activation of the brake system 18 can be allowed and can also be performed.

[0061] Similarly, in the case of a truck, two subclassifiers 24 for a passenger car and a truck may each provide a high probability 32 and the other subclassifier 24 may each provide a low probability 32. The obstacle classifier 28 may calculate a high final probability 34a based on these two high individual values ​​and using its feature vector 31. This may therefore still lead to activation of the brake system 18, even though the exact object type has not been clearly classified.

[0062] Example 2 from the table is an example classification of an object of a previously unknown class, for example a class not contained in the training data. The detailed classifiers 24 each provide a low probability 32 and the obstacle classifier 28 calculates a high final probability 34 c for the class “unknown”.

[0063] Example 3 from the table is an example of a classification result in the case of an unknown sensor fault. In this case, most of the detailed classifiers 24 provide a high probability 32 and the obstacle classifier 28 also calculates a high final probability 34c for the class “unknown” here.

[0064] Figure 3It is shown that the decomposition of the classification system 16 into two blocks 22, 28 can also be cascaded in order to enable an even more specialized and more robust classification. One, some or all of the first classifier 24 and / or the second classifier 28 can be formed by a first block 22' and a second block 26', respectively. In this case, the first block includes at least one first subclassifier 24', which calculates an intermediate probability 32', which is fed into the second subclassifier 28'. Similar to Figure 2 Specifically, the feature vectors 30, 31 can be fed into each of the first sub-classifiers 24' and into the second sub-classifier 28'. The second sub-classifier 28' calculates higher-level probabilities 32, 34 from the feature vectors 31 and the intermediate probabilities 32'.

[0065] In addition, it should be pointed out that "comprising" does not exclude any other elements or steps, and "one" does not exclude a plurality. In addition, it should be pointed out that the features or steps described with reference to the above embodiment can also be used in combination with other features or steps of other embodiments described above. The reference numerals in the claims should not be regarded as limiting.

Claims

1. A method for determining the state of the surrounding environment of a vehicle (10), the method include: - receiving sensor data (20) from at least one surroundings sensor (14) of the vehicle (10); - feeding at least one first portion (20a) of the sensor data (20) into at least one first classifier (24); - generating, by means of the first classifier (24), from a first portion (20a) of the sensor data (20) an intermediate probability (32) for a subclassification of a certain object type or a certain situation in the surroundings of the vehicle (10); - feeding at least one second portion (20b) of the sensor data and at least one intermediate probability (32) into a second classifier (28); - generating a final probability (34a, 34b, 34c) for the surrounding state from the second part (20b) of the sensor data (20) and the at least one intermediate probability (32) by means of the second classifier (28), wherein the surrounding state is described by the presence and / or absence of one or more superordinate categories of a subclassification of a certain object type or a certain situation in the surroundings of the vehicle (10).

2. The method according to claim 1, further comprising: include: - feeding at least a portion (20a) of the sensor data (20) into a plurality of first classifiers (24); - generating an intermediate probability (32) from the fed-in sensor data (20) by each of the first classifiers (24); - feeding the intermediate probability (32) into the second classifier (28).

3. The method according to claim 1 or 2, in, The first classifier (24) and the second classifier (28) determine corresponding probabilities (32, 34a, 34b, 34c) by means of different classification methods; and / or wherein the first classifier (24) and / or the second classifier (28) is based on a learning-based classification method, wherein the learning-based classification method is trained using classified training data; and / or The first classifier (24) and / or the second classifier (28) are based on a model-based classification method, wherein the model-based classification method includes a model that is built based on classified training data.

4. The method according to claim 1 or 2, further comprising: include: - determining a first feature vector (30) for the first classifier (24) from the sensor data (20), wherein the first classifier (24) determines the intermediate probability (32) from the first feature vector (30); - for the second classifier (28), a second feature vector (31) is determined from the sensor data (20), wherein the second classifier (28) determines the final probability (34a, 34b, 34c) from the second feature vector (31).

5. The method according to claim 4, in, The first eigenvector (30) and the second eigenvector (31) are different; and / or wherein a feature vector (30) is generated for each of the plurality of first classifiers (24) from the sensor data (20); and / or The feature vectors (30) of the plurality of first classifiers (24) are different.

6. The method according to claim 1 or 2, in, a first portion (20a) of the sensor data (20) used by the at least one first classifier (24) being different from a second portion (20b) of the sensor data (20) used by the second classifier (28); The first part (20a) and the second part (20b) are independent parts of the sensor data (20).

7. The method according to claim 1 or 2, in, The first classifier (24) and / or the second classifier (28) are each composed of at least one first sub-classifier (24') and one second sub-classifier (28'), wherein the probability (32') generated by the at least one first sub-classifier (24') is fed into the second sub-classifier (28').

8. The method according to claim 1 or 2, in, The at least one first classifier (24) generates more than one intermediate probability (32); and / or The second classifier (28) generates a first final probability (34a) of "whether the surrounding environment state exists", the second classifier (28) generates a second final probability (34b) of "whether the surrounding environment state does not exist", and the second classifier (28) generates a third final probability (34c) of "whether the sensor data (20) can be classified".

9. The method according to claim 1 or 2, further comprising: include: The final probability (34a, 34b, 34c) is output to a control system (18) of the vehicle (10), and the vehicle (10) is controlled based on the final probability (34a, 34b, 34c).

10. A system (16) for determining a state of the surrounding environment of a vehicle (10), the system (16) include: at least one first classifier (24) for receiving at least one first portion (20a) of sensor data (20) of the vehicle (10) and for generating, from the received sensor data (20), an intermediate probability (32) for a subclassification of a certain object type or a certain situation in the surroundings of the vehicle (10); - at least one second classifier (28) for receiving at least a second part (20b) of the sensor data (20) of the vehicle (10) and for generating a final probability (34a, 34b, 34c) about the surrounding state from the received sensor data (20) and the at least one intermediate probability (32), wherein the surrounding state is described by the presence and / or absence of one or more superordinate categories of a subclassification of a certain object type or a certain situation in the surroundings of the vehicle (10).

Images