Method for evaluating a method for controlling an at least partially automated mobile platform, evaluation device, computer program and storage medium

Abstract

A method for evaluating a first method for controlling an at least partially automated mobile platform in a surrounding of the mobile platform is proposed, the method having the following steps: determining a control action for a scenario of the surrounding by means of the first method; determining a confidence value for the control action determined by means of the first method; determining a representation of the scenario of the surrounding of the mobile platform if the determined confidence value is less than a trust value in order to evaluate the first method with the scenario. An evaluation device, a computer program and a machine-readable storage medium are also proposed.

Description

Technical Field

[0001] This invention relates to a method for evaluating a method for controlling a mobile platform that is at least partially automated. Background Technology

[0002] The automation of driving has emerged with the increasing availability and superior performance of sensor systems for detecting the surrounding environment in vehicles. Sensor data is fused into an environmental model to represent the vehicle's surroundings. The requirements for the scope and quality of this environmental model depend on the driving functions implemented upon it. In autonomous vehicles, complete driving decisions are made based on the environmental model, and actuators are manipulated accordingly.

[0003] The open context nature of road traffic conditions presents a significant challenge to the development of driver assistance systems and (partially) automated driving functions. When developing systems to represent the surrounding environment, it is often impossible to fully anticipate which combinations of road topology, traffic flow, weather, lighting, etc., will occur in reality, and which of these combinations will be particularly challenging for algorithms (e.g., algorithms for recognizing traffic signs or behavioral planning). This applies not only to classic model-based approaches but also to data-based approaches.

[0004] By incorporating expert knowledge or appropriately selecting training data, one attempts to design a representation of the surrounding environment that is sufficiently accurate in identifying important and relevant objects within the desired environment, or from which behavioral plans can be derived sufficiently accurately. However, during training or design, it may not be possible to consider situations or scenarios of the surrounding environment that are important for actual operation, or there may be a lack of suitable training data for certain situations or scenarios of the surrounding environment. Summary of the Invention

[0005] According to aspects of the invention, a method for evaluating a first method for controlling a mobile platform that is at least partially automated is provided, a method for providing control signals, an evaluation device, a computer program, and a machine-readable storage medium. Advantageous configurations are the subject of the following description.

[0006] Throughout the description of this invention, the order of the method steps is shown in such a manner that the method is easy to understand. However, those skilled in the art will recognize that multiple method steps can also be performed in different orders and result in the same or corresponding results. In this sense, the order of the method steps can be changed accordingly. Several features are provided with counters to improve readability or make the assignment more explicit, but this does not imply the existence of specific features.

[0007] According to one aspect of the present invention, a method is provided for evaluating a first method for controlling at least partially automated mobile platforms in the surrounding environment of a mobile platform, the method comprising the following steps:

[0008] In one step, a control action is determined for a scenario of the surrounding environment using a first method. In another step, a confidence value for the control action determined using the first method is determined. In yet another step, if the determined confidence value is less than a trust value, a representation of the scenario of the mobile platform's surrounding environment is determined so that the first method can be evaluated against that scenario.

[0009] A mobile platform can be understood as a mobile, at least partially automated system, and / or a driver assistance system. An example could be a vehicle that is at least partially automated or has a driver assistance system. That is to say, for this purpose, a system that is at least partially automated includes a mobile platform with at least partial automation functionality, but a mobile platform also includes vehicles and other mobile machines that include driver assistance systems.

[0010] The concept of evaluating the first method should be interpreted broadly and includes the evaluation, analysis, and improvement of the first method.

[0011] The scenario and its representation of the surrounding environment of the mobile platform, particularly including objects and their relative positions and / or orientations or velocities, are especially important for evaluating the first method. For example, vehicles at a greater distance from the vehicle (in which the first method is used in test mode) are less relevant to control actions (e.g., lane changes). Here, the scenario also includes location information, such as GPS location. For example, the location information can be used to check whether the control actions are determined by the first method with a sufficiently high confidence value in a scenario with a certain traffic technology surrounding the vehicle (e.g., highways and / or tunnels and / or intersections).

[0012] Alternatively or additionally, a map with lane information and / or traffic signs may be provided in digital form for this method, and / or such a digital map may be generated by this method. The method may then be configured to assign the location of the mobile platform and / or the corresponding location of important related objects to the geographic location of the digital map.

[0013] Advantageously, this method can be used for evaluation in order to publish (freizugeben) the method for practical use.

[0014] Especially when the first method used for control relates to a method used for behavior planning, this method can be used to evaluate the first method. This is because there is a problem with methods used for behavior planning where, when compared to, for example, established behavior planning methods, differences relative to the current situation can be easily determined. Thus, for example, compared to established behavior planning methods, the first method might suggest changing lanes to the left instead of going straight. Since the action proposed by the first method may have future consequences, this immediately determined difference is insufficient to evaluate the first method, because such instantaneous differences do not allow for any conclusions about the further development of the situation in the future.

[0015] For example, it is impossible to determine how the situation will develop in the event of a lane change, i.e., whether the situation will be advantageous or even lead to an accident.

[0016] The method described herein advantageously allows for the evaluation of the first method, especially when the method is executed using multiple vehicles. This is particularly true when the first method is run in test mode or shadow mode, especially when using a mobile platform, thus accelerating the development of the first method and also supporting release beargumentation.

[0017] In particular, this method can be used to evaluate the first method or function used for control, the results of which or their actions may have an impact in the future.

[0018] In particular, the first method can be run in test mode or shadow mode to collect information about which scenarios or conditions in the representation of the surrounding environment the first method (e.g., behavior planner) determines actions (e.g., control actions) with low confidence, so as to improve the first method for such scenarios, for example.

[0019] Here, the test mode or shadow mode can be a first method operating in passive mode, providing it with input data from the sensor system of the mobile platform or input data for further processing, wherein, in the test mode or shadow mode, the method is not used to control or manipulate the actuators of the mobile platform.

[0020] For example, improvements can be achieved by incorporating the following scenarios into the data structure of a data-based function in a first approach: in these scenarios, the control action is determined with low confidence values.

[0021] If, for example, the data-based function of the first method has never been trained with dense traffic but faces real-world traffic congestion scenarios, then this interpretation is used to determine low-confidence values ​​for control actions. With the described evaluation method, these scenarios can be identified in order to evaluate or (if necessary) improve the corresponding first method. For this purpose, the corresponding scenarios can be abstracted from a large number of such scenarios and considered in the further development of the first method (e.g., in the training of the behavioral policy).

[0022] In other words, this method can be used to evaluate a first method for determining control actions by: determining the corresponding confidence value of the control action using a method for determining confidence values, and collecting representations of scenarios in test mode where the control action is determined with a confidence value less than the confidence value.

[0023] For example, a confidence value can be determined, wherein the method used to determine the confidence value is able to identify whether the confidence value is in the extrapolation range (i.e., a sufficiently similar scenario or situation is unknown from the training or manual specification phase) or in the interpolation range (i.e., a sufficiently similar scenario or situation is known from the development phase) with respect to the applied input data (i.e., a representation of the surrounding environment).

[0024] Multiple such scenarios can be selected on various mobile platforms, for example, based on the confidence estimate, in order to minimize the number of scenario representations to be transmitted and the associated control actions or confidence values, and to transmit them wirelessly and / or wiredly and / or in conjunction with a data carrier to a central authority (e.g., the cloud) for evaluation.

[0025] Therefore, a first method based on expert knowledge and / or data-driven implementation using training data can be introduced into vehicles or fleets during training mode or shadow mode operation.

[0026] Corresponding to the desired evaluation, filtering criteria can be defined for the transmission of scenarios. These criteria, for example, describe scenarios or situations for which there are no sufficiently good corresponding data in the training data. For instance, scenarios can be identified in the form of location data (GPS location) and / or driving trajectories of the vehicle and surrounding traffic. Such filtered scenarios can be transmitted along with other feature parameters (such as corresponding confidence values ​​and / or control actions) to a central agency (cloud) for evaluation.

[0027] For example, scene representations collected in this way can be used to define clear rules regarding classic expert-knowledge-based methods: how to handle situations that are currently unknown. Alternatively or additionally, the corresponding scene representations can be reconstructed in simulations and / or used to train data-based methods. An improved version of the first method can then be generalized again in training mode on a fleet of vehicles for further evaluation, so that Y provides important arguments for the release of the first method if reliability is sufficient and / or the method can be re-implemented if the evaluation results are not good enough.

[0028] Alternatively or additionally, the first method can also be used for offline analysis when not running in test mode. For this purpose, large amounts of data can be increased and stored for different scenarios in various possible surrounding environments for the mobile platform. It is important to fully consider the scenarios or situations that occur in reality.

[0029] When using the first method in test mode or shadow mode, it is advantageous to make informed decisions by filtering the scenes to be transmitted: which scenes or situations are particularly important and relevant, so that only a subset needs to be transmitted.

[0030] Therefore, by means of the method proposed herein for evaluating the first method for control, unknown driving conditions in test mode or shadow mode can be identified in the form of scenarios, so as to evaluate the first method for behavior planning in particular.

[0031] According to one perspective, a first method for controlling a platform that is at least partially automated is a method for behavioral planning of a mobile platform that is at least partially automated.

[0032] In particular, for the first approach involving behavioral planning, the described method for evaluation can be used advantageously because behavioral planning involves actions that have future impacts and therefore can only be adequately characterized for evaluation through immediate comparison with other methods.

[0033] Here, the behavior planner can be understood as a method involving an initial stage of trajectory planning, in which decisions are made regarding the future behavior of the mobile platform, such as making a lane-changing decision, in response to traffic conditions / scenarios in the surrounding environment. The behavior planner can also be understood, alternatively or additionally, as a method for providing trajectories. To this end, the behavior planner utilizes deterministic parameters related to the surrounding environment as input variables. (These parameters are determined with the aid of a sensor system) The main objects in the environment surrounding the mobile platform and their relative arrangement and / or orientation relative to each other and relative to the mobile platform are obtained in the form of a scene representation of the environment surrounding the mobile platform.

[0034] The determination parameters of a sensor system that are related to its surrounding environment are determined as follows: these determination parameters are related to the surrounding environment of the sensor system and are determined using data from one or more sensor systems.

[0035] Here, the parameters related to the surrounding environment can be parameters that are obtained by means of sensor system data about the target being measured, and by analysis and / or summarization to represent the surrounding environment of the sensor system.

[0036] For example, L-shaped analysis processing of image segmentation, stereo pixels, or LiDAR systems is used to measure target (object detection), such as to identify, measure object categories like cars, and determine their locations.

[0037] Here, the defining parameters related to the surrounding environment can be more abstracted than the pure data of the sensor system. For example, defining parameters related to the surrounding environment can include objects, features, stereo pixels, the size (Ausmaβe) of the corresponding defined object, object type, three-dimensional “bounding box”, object category, such as L-shapes and / or edges and / or reflection points in a LiDAR system.

[0038] Here, the parameters related to the surrounding environment can also include data from the sensor system and / or a list of objects in the environment surrounding the mobile platform.

[0039] According to one aspect, a first method is evaluated using multiple at least partially automated mobile platforms, and / or representations of corresponding scenarios from a portion of the multiple at least partially automated mobile platforms are wirelessly transmitted to a central agency for evaluating the first method.

[0040] By leveraging multiple, at least partially automated, mobile platforms, this method can be extended to a fleet of vehicles, enabling the acquisition of substantial field knowledge about the first method in a relatively short period. This allows for reliable release decisions or targeted further development of the first method based on such evaluation.

[0041] According to one aspect, only a portion of the representation of the corresponding scenario is transmitted to the central agency for evaluation, and this portion depends on the representation of the corresponding scenario and / or the first method, in order to minimize the amount of data to be transmitted.

[0042] Advantageously, since only a portion of the representation of the corresponding scenario is transmitted to the central authority, it is possible to determine at the mobile platform containing scenario-defined data which condition descriptions or scenario representations should be transmitted to the central authority for evaluation. Here, before transmitting the scenario representation to be transmitted, the scenario representation to be transmitted can be selected based on scenarios that are important to the evaluation of the first method for control.

[0043] According to one aspect, the corresponding control actions of the corresponding at least partially automated mobile platform are transmitted.

[0044] Advantageously, by transmitting corresponding control actions determined in a specific scenario of the mobile platform's surrounding environment, the first method for control can be evaluated using multiple control actions. Alternatively or additionally, control actions of the vehicle driver can also be transmitted when the first and / or second methods are not activated.

[0045] According to one aspect, in a corresponding at least partially automated mobile platform, the first method is run in test mode.

[0046] Therefore, the first method can be evaluated using real-world conditions at an earlier stage of its development.

[0047] For example, this leads to the possibility of comparing the performance of the new first method with that of the current method and / or the driver of the mobile platform. The collected data (e.g., especially scene representations) is then identified and stored and / or transmitted to the cloud or a central authority for evaluation.

[0048] According to one aspect, a second method is used to determine control actions, and the second method at least partially controls a mobile platform that is at least partially automated to evaluate the first method.

[0049] One approach proposes determining confidence values ​​by comparing control actions determined using a first method with control actions determined using a second method from the same scenario.

[0050] Since the second method at least partially controls the mobile platform, a good basis for comparison can be derived for evaluating the first method, because the scenario of the mobile platform's surrounding environment can be the same for both methods in terms of determining the control actions.

[0051] According to one aspect, confidence values ​​are determined additionally or alternatively using the self-evaluation of the first method.

[0052] According to one aspect, confidence values ​​are determined by comparing the control actions determined using a first method with control actions from a vehicle driver on a mobile platform that is at least partially automated and operating in the same scenario.

[0053] Advantageously, this leads to the possibility that even if a second method for at least partially controlling a mobile platform has not yet been released for road traffic, it is possible to perform a comparison with the behavior of the vehicle driver of the mobile platform.

[0054] One approach proposes using machine learning methods to determine confidence values.

[0055] Here, examples of machine learning methods are (Bayesian) neural networks, (where necessary) combined with fully connected neural networks, (where necessary) using classic regularization and stabilization layers (e.g., batch normalization and training-drop-outs), using different activation functions (e.g., Sigmoid and ReLU), such as support vector machines, boosting methods, decision trees, Gaussian processes (especially with variance calculation for prediction), and random forests.

[0056] One approach proposes using a model-based method to determine confidence values.

[0057] This model-based approach can be generated with the help of expert knowledge, and the confidence value can be determined based on the following: the model-based approach can identify whether the current input data (i.e., especially the representation of the surrounding environment) is within the extrapolation range of the method (i.e., for the model-based approach, there are not enough similar situations known from the training or manual specification phase) or within the interpolation range (i.e., for the model-based approach, there are enough similar situations from the development phase).

[0058] A method is proposed that provides control signals for operating a vehicle that is at least partially automated, based on control actions determined by a first method and determined by one of the aforementioned methods; and / or provides warning signals for warning vehicle occupants based on control actions determined by the first method.

[0059] The term "based on" should be interpreted broadly with respect to the characteristic that the control signal is provided based on the control action determined by the first method. This term should be understood as meaning that the control action determined by the first method can take into account any determination or calculation used for the control signal, whereby this does not exclude the possibility of considering other input parameters for the determination of the control signal. This applies accordingly to the provision of warning signals.

[0060] For example, highly automated systems can use such control signals to transition to a safe state, such as by performing a slow stop on the shoulder in at least partially automated vehicles.

[0061] An evaluation device is proposed, which is configured to perform one of the methods described above.

[0062] With the help of this evaluation device, the method can be easily introduced into different mobile platforms.

[0063] According to one aspect, a computer program is described that contains instructions, which, when executed by a computer, cause the computer to perform one of the methods described above. This computer program enables the use of the methods in various systems.

[0064] A machine-readable storage medium is described, on which the aforementioned computer program is stored. Using this machine-readable storage medium, the aforementioned computer program is transportable. Attached Figure Description

[0065] Reference Figure 1 Embodiments of the present invention are shown and will be described in more detail below.

[0066] The attached diagram shows:

[0067] Figure 1 A data flow diagram is shown for evaluating a first method for controlling a mobile platform that is at least partially automated. Detailed Implementation

[0068] Figure 1 The data flow of a method 100 for evaluating a first method for controlling at least partially automated mobile platform 200 in its surrounding environment 110 is schematically depicted. A representation of the surrounding environment 110 can be generated from the surrounding environment 110 of mobile platform 200 using sensor 120.

[0069] The first method can be run in test mode for evaluation without having a direct impact on the control of the mobile platform 200. Here, the mobile platform 200 can be controlled at least partially by the second method.

[0070] In step S1, the control action is determined based on the scene of the surrounding environment using the first method.

[0071] In step S2, a confidence value for the control action is determined using the first method.

[0072] Here, the confidence value can be determined as follows: by comparing the control action determined by the first method with the control action determined by the second method from the same scenario, and additionally or alternatively by comparing the control action determined by the first method with the control action of a vehicle driver from the same scenario, at least partially automated mobile platform, and additionally or alternatively by using a machine learning system, and additionally or alternatively by using a model-based method, or additionally or alternatively by self-evaluation of the first method.

[0073] In step S3, if the determined confidence value is less than the trust value, a representation of the scene of the surrounding environment 110 of the mobile platform 200 is determined so as to evaluate the first method in that scene.

[0074] In step S4, a filter can be made to determine whether to transmit a representation of a scenario to the central agency 170 for evaluation in a scenario where the confidence value used to determine the control action is less than a trust value. That is, only a portion of the representation of each respective scenario is transmitted to the central agency 170 for evaluation. This portion of the transmitted representation of a given scenario may depend on the representation of the given scenario and / or the first method, in such a way that only the representation of the scenario necessary for evaluation by the first method is transmitted, thereby minimizing the amount of data to be transmitted.

[0075] In step S5, the corresponding representation of the scene to be transmitted can be transmitted to the central agency 170.

[0076] This method can be executed using multiple vehicles or mobile platforms 190, and in the corresponding step S7, the data is transmitted to a central agency 170 for evaluation. This transmission of representations of corresponding scenes for corresponding control actions on the corresponding vehicles or mobile platforms 200 can be wirelessly transmitted from the multiple vehicles or mobile platforms 190 to the central agency 170.

[0077] In step S6 of the method, a first method for controlling at least partially automated mobile platforms can be evaluated using multiple representations of scenarios from multiple mobile platforms and corresponding control actions. Here, the first method can be a method for behavior planning of at least partially automated mobile platforms.

Claims

1. A method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) surrounding a mobile platform (200) that is at least partially automated, wherein, In a corresponding at least partially automated mobile platform (200), the first method is run in a test mode, wherein, in the test mode, the first method is not used to control or manipulate the actuators of the mobile platform (200), and comprises the following steps: The first method is used to determine the control action based on the scene of the surrounding environment; The confidence value of the control action is determined using the first method, wherein the confidence value of the control action is determined using a method for determining the confidence value, wherein the method for determining the confidence value is able to identify whether the representation of the confidence value with respect to the surrounding environment is within the extrapolation range or within the interpolation range, wherein if the representation of the confidence value with respect to the surrounding environment is within the extrapolation range, then a scene or situation sufficiently similar to the scene is unknown from the training or manual specification phase, wherein if the representation of the confidence value with respect to the surrounding environment is within the interpolation range, then a scene or situation sufficiently similar to the scene is known from the development phase; If the determined confidence value is less than the trust value, a representation of the scene of the surrounding environment (110) of the mobile platform (200) is determined in order to evaluate the first method in the scene.

2. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) of at least partially automated mobile platform (200) according to claim 1, wherein, A first method for controlling a mobile platform (200) that is at least partially automated is a method for planning the behavior of the mobile platform (200) that is at least partially automated.

3. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) surrounding a mobile platform (200) that is at least partially automated, according to claim 1 or 2, wherein, The first method is evaluated using multiple at least partially automated mobile platforms, and / or representations of corresponding scenarios from a portion of the multiple at least partially automated mobile platforms are wirelessly transmitted to a central agency (170) for evaluation of the first method.

4. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) of at least partially automated mobile platform (200) according to claim 3, wherein, Only a portion of the representation of the corresponding scene is transmitted to the central agency (170) for evaluation, and the portion depends on the representation of the corresponding scene and / or the first method, in order to minimize the amount of data to be transmitted.

5. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) of at least partially automated mobile platform (200) according to claim 3, wherein, The corresponding control actions of the at least partially automated mobile platform (200) are transmitted.

6. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) surrounding a mobile platform (200) that is at least partially automated, according to claim 1 or 2, wherein, The control action is determined by a second method, and the second method at least partially controls the at least partially automated mobile platform (200) for evaluating the first method.

7. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) surrounding a mobile platform (200) that is at least partially automated, according to claim 1 or 2, wherein, The confidence value is determined by comparing the control action determined by the first method with the control action determined by the second method from the same scenario.

8. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) surrounding a mobile platform (200) that is at least partially automated, according to claim 1 or 2, wherein, The confidence value is determined by comparing the control action determined by the first method with the control action of the vehicle driver from the same scenario, which is at least partially automated mobile platform (200).

9. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) surrounding a mobile platform (200) that is at least partially automated, according to claim 1 or 2, wherein, The confidence value is determined using machine learning methods.

10. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) surrounding a mobile platform (200) that is at least partially automated, according to claim 1 or 2, wherein, The confidence value is determined using a model-based method.

11. The method (100) for evaluating a first method for controlling a mobile platform (200) in an environment (110) surrounding a mobile platform (200) that is at least partially automated, according to claim 1 or 2, wherein, Based on the control actions determined by the first method, control signals are provided for operating a partially automated mobile platform (200) configured as at least partially automated vehicles; and / or, based on the control actions determined by the first method, warning signals are provided for warning vehicle occupants.

12. An evaluation apparatus configured to perform the method according to any one of claims 1 to 11.

13. A computer program product comprising instructions that, when implemented by a computer, cause the computer to perform the method according to any one of claims 1 to 11.

14. A machine-readable storage medium on which a computer program product according to claim 13 is stored.

Images