Method for providing machine learning control function for vehicle control based on provided vehicle sensor data

Abstract

A method for providing a machine-learned control function for vehicle control based on provided vehicle sensor data, comprising the method steps of: G) providing a control function based on an artificial neural network including at least one outputtable control command; H) providing an evaluation catalog for evaluating driving scenarios; I) providing a training dataset including driving scenarios; J) training the control function with the training dataset; K) applying the control function to a test dataset including driving scenarios and evaluating the driving scenarios using the evaluation catalog; and L) expanding the training dataset and / or the test dataset, or reducing or increasing the complexity of the trained control function.

Description

[Technical field]

[0001] The present invention relates to a method for providing machine-learned control functions for vehicle control based on provided vehicle sensor data. [Background technology]

[0002] Currently, vehicles have a control unit that executes control functions defined by the system developer. In this case, the vehicle is equipped with a number of sensors that provide vehicle sensor data for the control functions, and the control functions generate control commands based on these vehicle sensor data. The driver of the vehicle often does not have a direct influence on the control commands generated by the control functions. Examples of control functions are, for example, cruise control, distance control, or lane change warning systems, which utilize vehicle sensor data from radar, lidar, or cameras.

[0003] A known method of providing a control function for vehicle control involves providing outputtable control commands by a deterministic / algorithmically developed computer program implemented in a control unit of the vehicle, where the outputtable control commands are defined by the computer program developer in such a way that the control commands are explicitly assigned to predefined scenarios. This method of providing a control function for vehicle control is not suitable for providing complex control functions, since reality provides complex driving scenarios, which the computer program developer must take into account in order to provide a control function that does not output risk-inducing control commands. However, this is only feasible to a very limited extent and for only a small number of driving scenarios, so that the control functions provided in this way are prone to erroneous reactions under real conditions, especially to unknown driving situations.

[0004] Another method of providing control functions involves providing control functions based on artificial neural networks, where the artificial neural networks are trained on training data sets representing as many driving scenarios as possible with different classifications, and autonomously derive rules for executing defined control commands. This allows providing complex control functions, since concrete rules do not need to be implemented in a computer program product. Nevertheless, this method of providing machine-learned control functions for vehicle control is prone to errors, since the artificial neural networks involved generate outputs with low reliability or large statistical uncertainty for scenarios that are not or poorly represented by the training data sets.

[0005] German patent applications DE 102018116036 A1 and DE 102017006434 A1 are known from the prior art. Summary of the Invention [Problem to be solved by the invention]

[0006] The object of the present invention is to overcome the drawbacks of the prior art and to propose a method for providing machine-learned control functions for vehicle control based on provided vehicle sensor data. [Means for solving the problem]

[0007] The present invention provides a method for providing a machine-learned control function for vehicle control based on provided vehicle sensor data, the method comprising: A) providing a control function based on an artificial neural network (of a machine learning component) including at least one outputtable control command; B) a method step of providing an evaluation catalog for evaluating driving scenarios; C) providing a training data set including driving scenarios; D) A method step of training a control function on a training data set; E) Apply the control functions to a test dataset including driving scenarios and use the evaluation catalog to Control function applied A method for evaluating a driving scenario; F) Expanding the training and / or test datasets By increasing the number of datasets, - a method step of reducing or increasing the complexity of the trained control function; repeating method steps D), E) and F) iteratively using the expanded training data set, the test data set and the control function; of Contains fruit, The method steps are repeated iteratively until the driving scenario is evaluated as not inducing a danger using the evaluation catalogue. .

[0008] A control function controls the functions of the vehicle with at least one control command. A control function is any function that affects the control of the vehicle, whether that be the provision of information or specific instructions for action.

[0009] A control command refers to a specific action instruction and / or signal to trigger a control function to be automatically executed. The artificial neural network may be implemented in a control unit of the vehicle or in a server outside the vehicle and interacts with the control unit of the vehicle.

[0010] The evaluation catalogue can be designed to classify driving scenarios into known or unknown scenarios, non-hazardous or hazardous scenarios, safe or unsafe scenarios, and evaluate them accordingly.

[0011] An unknown driving scenario is a driving scenario that is not represented in the training dataset. An unknown driving scenario is a driving scenario that was not considered by the developer and was not included in the evaluation catalog and / or training dataset.

[0012] An unsafe driving scenario is one in which the artificial neural network produces outputs that are unreliable or have large statistical uncertainties. An unsafe driving scenario may result in control commands that pose a risk to the control function.

[0013] The training data set may represent a simulated run of the vehicle. A reduction in the complexity of the control function may occur, for example, if the control function runs too slowly on the vehicle's control unit or if the control function itself has too large memory requirements. A reduction in the complexity of the control function leads to a reduction or avoidance of overfitting by the artificial neural network, improving the control function. An increase in the complexity of the control function may occur if the control function cannot be adequately adapted (trained) to the run scenarios provided in the training data set, and may manifest itself, for example, in an incorrect behavior of the control function.

[0014] According to a preferred embodiment, the method comprises the method step of iteratively repeating method steps D) and / or E) and / or F) using extended training and / or test data sets and / or control functions of reduced or increased complexity, said iterative repetitions being able to be performed, for example, until all driving scenarios have been evaluated as not inducing a risk using the evaluation catalogue.

[0015] According to another preferred aspect, evaluating the driving scenario using the evaluation catalog includes determining an unknown driving scenario.

[0016] According to a particularly preferred embodiment, evaluating the driving scenarios using the evaluation catalogue comprises determining unsafe driving scenarios.

[0017] According to an advantageous embodiment, the extension of the training data set and / or the test data set is based on driving scenarios that are assessed as unknown, which allows a continuous and adaptive extension of the training data set and training of the control functions to driving scenarios that are relevant for increased safety.

[0018] According to another advantageous aspect, the extension of the training data set and / or the test data set is performed on the basis of driving scenarios that are assessed as unsafe, which allows a continuous and adaptive extension and training of the control functions in order to avoid risk-inducing control commands.

[0019] A control command is based on the output of an artificial neural network and is said to be safe if the output of the artificial neural network is carried out with sufficient reliability or statistical certainty.

[0020] According to a particularly advantageous aspect, the method comprises a method step of analytically validating the control function taking into account defined value ranges of a parameter, which may relate to vehicle sensor data and / or data derived from vehicle sensor data. Methods of analytical validation are known methods, such as for example "Reluplex" or "ReluVal".

[0021] According to a particularly advantageous aspect, the method comprises the method step of probabilistic verification of the control functions, which means a data-driven evaluation and optimization of the test case coverage, which means the quantification by inductive and deductive analysis of the rules learned by the artificial neural network.

[0022] According to an advantageous aspect, the reduction in the complexity of the control function comprises a simplification of the artificial neural network, which may be performed in order to lower the computational and / or memory resources required for the artificial neural network.

[0023] According to another advantageous aspect, the simplification comprises a reduction of neurons and / or layers of the artificial neural network.

[0024] According to a particularly advantageous aspect, the reduction in the complexity of the control function comprises a reduction in the vehicle sensor data provided, where the reduction may be a reduction in the frequency with which the vehicle sensor data is provided.

[0025] According to a particularly advantageous embodiment, the design of the computational and storage capacities in the control unit of the vehicle is carried out on the basis of an artificial neural network.

[0026] According to a preferred embodiment, the design (complexity) of the artificial neural network is based on the computational and memory capacities in the vehicle's control unit.

[0027] According to another preferred aspect, the control function is provided in a vehicle and tested while in operation, which may be a real vehicle or a virtual / simulated vehicle, where the control function may be tested while in operation by simulating (generating) output control commands which are not actually executed by the vehicle.

[0028] According to a particularly preferred embodiment, the control function is provided in a vehicle, and the training data set and / or the test data set are expanded based on driving scenarios provided during (or by) the drive. The vehicle may be a real vehicle or a virtual / simulated vehicle, whereby the driving scenarios provided for the expansion of the training data set during the drive can be evaluated based on the evaluation catalogue.

[0029] The present invention includes a vehicle having a controller that performs the control functions provided in accordance with the method.

[0030] The present invention will now be described in more detail with reference to examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] As an example according to the present invention, a method for providing a machine learning control function for vehicle control based on provided vehicle sensor data is described, where the control function is a lane change warning system based on an artificial neural network for predicting lane change maneuvers of surrounding vehicles (LCMP, English: lane change maneuver predictor), where the LCMP is provided with vehicle sensor data from radar and lidar sensors and a camera.

[0032] The method includes the following steps. A) An artificial neural network-based control function, in this example a lane change warning system, for predicting lane change maneuvers (LCMP) of surrounding vehicles, including outputtable control commands is provided, where the control commands can be output based on LCMP that can predict "lane change right", "lane change left" and "no lane change" of surrounding vehicles. B) Providing an evaluation catalogue for evaluating driving scenarios. The evaluation catalogue defines driving scenarios as known or unknown, safe or unsafe, based on the implemented verification and validation objectives. Unsafe driving scenarios are those for which the LCMP produces outputs with low confidence or high statistical uncertainty. To check safety, potentially hazardous driving scenarios can be used, referencing criteria from the evaluation catalogue, e.g. minimum distance between surrounding vehicles. C) Providing a training data set including driving scenarios The training data set may include, for example, simulations of driving conditions. D) Training the control function on the training data set, where an artificial neural network is trained using known methods. E) applying the control function to a test data set comprising driving scenarios and evaluating the driving scenarios based on the evaluation catalogue. The test data set may, for example, include simulations of driving situations, in particular driving situations not represented by the training data set. Evaluation of the driving scenarios using the evaluation catalogue may include classifying the driving scenarios into safe or unsafe, known or unknown driving scenarios. F) Expand the training and / or test data sets or reduce or increase the complexity of the trained control functions. If unsafe driving scenarios are found during the evaluation, the training data set is expanded.

[0033] Taking LCMP as an example, the reduction of complexity can be achieved by reducing the number of layers of the artificial neural network and / or by reducing the number of neurons per layer. The increase of complexity can be achieved by increasing the number of layers of the artificial neural network and / or by increasing the number of neurons per layer. To increase or decrease the complexity of the trained control function, the initial artificial neural network can be replaced by an artificial neural network with reduced / increased complexity. Here, for example, the artificial neural network with reduced / increased complexity can be retrained to the provided training data set, without or with the aid of the initial artificial neural network.

[0034] Method steps D) and / or E) and / or F) are repeated iteratively, after each iteration using an extended training data set and / or a test data set in F) and / or using a control function of reduced or increased complexity in F), thereby ensuring that the training data set is continuously optimized for training the control function, reducing the number of unsafe driving scenarios and increasing the number of safe driving scenarios, improving the control function.

[0035] Here, reducing the complexity of the control function includes simplifying the artificial neural network by reducing the number of neurons and layers in the artificial neural network. Additionally, reducing the complexity of the control function includes reducing the vehicle sensor data provided. For example, in a lane change warning system, data from radar and lidar sensors and cameras is provided less frequently.

[0036] The evaluation of driving scenarios using the evaluation catalog includes determining unknown driving scenarios, where unknown driving scenarios are driving scenarios that are not represented or are poorly represented by the training data set, where the unknown driving scenarios can be generated accurately.

[0037] An example of a method for generating unknown driving scenarios reliably is "DeepXplore". Here, the input data for the artificial neural network is modified by gradient methods in a way that forces misclassification by the artificial neural network. As a result, new driving scenarios previously unknown to the artificial neural network are systematically generated. This process is performed under the additional condition that the neuron coverage increases continuously. The neuron coverage indicates how many neurons are activated after passing through the test data set, and is measured relative to the sum of all neurons in the artificial neural network. Here, a neuron is considered activated if its activation value exceeds a predefined threshold.

[0038] The evaluation of the driving scenarios based on the evaluation catalog includes determining unsafe driving scenarios. In the example given, unsafe driving scenarios are scenarios for which the LCMP produces outputs with low confidence or high statistical uncertainty. Unsafe driving scenarios can be generated with precision.

[0039] One example of a method for generating risk-inducing driving scenarios is the gradient method "Fast Gradient Sign". With this method, input data from driving scenarios are manipulated in a precise way to mispredict the neural network, and risk-inducing driving situations are systematically discovered. The resulting driving scenarios represent new driving scenarios in which the neural network produces unsafe outputs.

[0040] Applied to the example of a lane change warning system based on an artificial neural network for prediction of lane change maneuvers of surrounding vehicles (LCMP), this means that such driving scenarios represent lane change maneuvers that are incorrectly evaluated by the LCMP (e.g. a lane change is predicted by the LCMP even though no lane change is made) and therefore the lane change warning system will output hazard-inducing control commands.

[0041] The training and test data sets are extended based on unknown or assessed as unsafe driving scenarios, whereupon method steps D), E), F) are repeated iteratively, where for each iteration an extended training and test data set is used and / or a reduced complexity control function is used in F).

[0042] The method also includes a method step of analytically validating the control function taking into account defined value ranges of the parameters, which relate to the vehicle sensor data and to data derived from the vehicle sensor data. The analytical validation of the control function taking into account the defined value ranges of the parameters is based on parameterizable driving scenarios. In the example of an artificial neural network based lane change warning system for prediction of lane change maneuvers of surrounding vehicles (LCMP), the parameterizable driving scenarios are:

[0043] The controlled vehicle is traveling in the center lane of a three-lane highway, and other vehicles are moving from the right lane towards the center lane with a longitudinal speed of 100km / h - 120km / h and a lateral speed of 2km / h - 4km / h with a time interval of 1 second. The speeds and times specified here define the defined value ranges of the parameters.

[0044] For analytical validation of this scenario, it must be ensured that the LCMP outputs a hazard-inducing control command for a driving scenario that fits within this value range, and as a result, the lane change warning system outputs a hazard-inducing control command.

[0045] The analytical validation can be carried out using known methods such as "Reluplex" or "ReluVal". The advantage of this method step is that it determines whether the artificial neural network of the control function leads to outputs that do not result in hazard-inducing situations in driving scenarios characterized by values ​​in the range of values ​​of the considered parameters.

[0046] The design of the computational and memory capacities of the control unit of the vehicle is, in the considered example, based on an artificial neural network, whereby in particular the central computational unit and the main memory of the control unit are appropriately designed.

[0047] In a further method step, the trained control function is installed in the vehicle and tested during driving, where driving scenarios provided during the drive can extend the training data set and / or the test data set and the previous method steps can be repeated. [Configuration 1] 1. A method for providing a machine-learned control function for vehicle control based on provided vehicle sensor data, the method comprising: A) providing a control function based on an artificial neural network including at least one outputtable control command; B) a method step of providing an evaluation catalog for evaluating driving scenarios; C) providing a training data set including driving scenarios; D) a method step of training said control function with said training data set; E) applying said control functions to a test data set including driving scenarios and evaluating said driving scenarios using said evaluation catalogue; F) extending the training data set and / or the test data set or reducing or increasing the complexity of the trained control function; The method includes: [Configuration 2] The method of claim 1, further comprising a method step of iteratively repeating method steps D) and / or E) and / or F) using the expanded training data set and / or test data set and / or the reduced or increased complexity control function. [Configuration 3] 3. The method of claim 1, wherein evaluating the driving scenario using the evaluation catalog includes determining an unknown driving scenario. [Configuration 4] 4. The method of claim 1, wherein evaluating the driving scenarios using the evaluation catalogue includes determining unsafe driving scenarios. [Configuration 5] 4. The method of claim 3, wherein the expansion of the training data set and / or the test data set is performed based on the driving scenarios that are assessed to be unknown. [Configuration 6] 5. The method of claim 4, wherein the extension of the training data set and / or the test data set is based on the driving scenarios that are assessed as unsafe. [Configuration 7] 7. A method according to any one of the preceding claims, comprising the method step of analytically validating the control function taking into account defined value ranges of a parameter. [Configuration 8] 8. A method according to any one of the preceding claims, comprising a method step of probabilistic validation of the control function. [Configuration 9] 9. The method of any one of the preceding claims, wherein the reduction in the complexity of the control function comprises a simplification of the artificial neural network. [Configuration 10] 10. The method of claim 9, wherein the simplification comprises reducing neurons and / or layers of the artificial neural network. [Configuration 11] 11. The method of any one of the preceding claims, wherein the reduction in the complexity of the control function comprises a reduction in the vehicle sensor data provided. [Configuration 12] 12. The method according to any one of the preceding claims, wherein the design of computational and memory capacities in a control unit of a vehicle is based on said artificial neural network. [Configuration 13] 13. The method according to any one of the preceding claims, wherein the design of the artificial neural network is based on the computational and memory capacities in a control unit of the vehicle. [Configuration 14] 14. The method of any one of the preceding configurations, wherein the control function is provided in a vehicle and tested while in operation. [Configuration 15] 15. The method of claim 14, wherein the control function is provided in a vehicle, and the training data set and / or the test data set are expanded based on driving scenarios provided during driving. [Configuration 16] A vehicle having a control device that executes a control function provided by the method according to any one of claims 1 to 15.

Claims

1. 1. A method for providing a machine-learned control function for vehicle control based on provided vehicle sensor data, the method comprising: A) providing a control function based on an artificial neural network including at least one outputtable control command; B) a method step of providing an evaluation catalog for evaluating driving scenarios; C) providing a training data set comprising said driving scenario; D) a method step of training said control function with said training data set; E) applying the control function to a test data set including the driving scenario and evaluating the driving scenario to which the control function has been applied using the evaluation catalogue; F) increasing the number of data sets by expanding the training data set and / or the test data set or decreasing or increasing the complexity of the trained control function; - iteratively repeating method steps D), E) and F) using the expanded training data set, the test data set and the control function; Including, the method steps of iteratively repeating are performed until the driving scenario is evaluated as not inducing a risk using the evaluation catalogue. method.

2. The method of claim 1 , wherein the evaluating the driving scenario using the evaluation catalog includes determining an unknown driving scenario.

3. The method of claim 1 or 2, wherein said evaluating said driving scenarios using said evaluation catalogue comprises determining unsafe driving scenarios.

4. The method of claim 2 , wherein the extension of the training data set and / or the test data set is performed based on the driving scenarios being assessed as unknown.

5. The method of claim 3 , wherein the extension of the training data set and / or the test data set is based on the driving scenarios being assessed as unsafe.

6. - analytically validating said control function taking into account defined value ranges of parameters, the parameter value range relates to the vehicle sensor data and / or data derived from the vehicle sensor data; 6. The method according to any one of claims 1 to 5.

7. 7. A method according to any one of the preceding claims, comprising the method step of probabilistic validation of the control functions.

8. 8. The method of claim 1, wherein the reduction in the complexity of the control function comprises a simplification of the artificial neural network.

9. The method of claim 8 , wherein the simplification comprises a reduction of neurons and / or layers of the artificial neural network.

10. 10. The method of claim 1, wherein the reduction in the complexity of the control function comprises a reduction in vehicle sensor data provided.

11. 11. The method according to claim 1, wherein the design of the computational and memory capacities in a control unit of a vehicle is based on said artificial neural network.

12. 12. The method according to any one of the preceding claims, wherein the design of the artificial neural network is based on the computational and storage capacities in a control unit of the vehicle.

13. 13. A method according to any preceding claim, wherein the control functions are provided in a vehicle and tested whilst in operation.

14. The method of claim 13 , wherein the control function is provided in a vehicle, and the training and / or test data sets are expanded based on driving scenarios provided during a drive.

15. A vehicle having a control device which performs a control function provided by a method according to any one of claims 1 to 14.