CONFIGURATION OF A CONTROL SYSTEM FOR A AT LEAST PARTIALLY AUTONOMOUS MOTOR VEHICLE

Description

[0001] The present invention relates to a method, a computer program with instructions, and a device for configuring a control system for an at least partially autonomous motor vehicle. The invention further relates to a motor vehicle that uses such a method or such a device.

[0002] Highly automated driving relies on a safe, reliable, and real-time capable vehicle system. To achieve this, it is essential to perform key functions redundantly for fail-safe operation. This involves utilizing multiple resources within the vehicle, without actually improving the functionality itself. The more resources these functions require, the greater the cost-benefit gap becomes. This is particularly true for functions based on artificial intelligence (AI), such as AI functions based on deep learning.

[0003] Against this background, DE 10 2017 006 599 A1 describes a method for operating a motor vehicle capable of at least partial automation. In this method, at least three artificial neural networks are trained independently of one another during disjoint training drives of the motor vehicle, each using an end-to-end approach. This training is based on training actuator data from a vehicle actuator system recorded during the training drives and on training sensor data from a vehicle sensor system correlated with the training actuator data. During at least partially automated operation of the motor vehicle, actual sensor data is recorded as input data for the neural networks, and training actuator data is assigned to the actual sensor data as output data for the neural networks based on a comparison with the training sensor data.The training actuator data from all neural networks is fed into a fusion module, which fuses the training actuator data from all neural networks according to a predefined rule and determines actual actuator data based on the result of the fusion. Using this determined actual actuator data, the vehicle's lateral and / or longitudinal control is performed, at least partially automatically.

[0004] Furthermore, DE 10 2017 107 837 A1 describes an adaptable sensor arrangement. The sensor arrangement comprises at least one sensor element with a control and evaluation unit. The sensor data is evaluated using a classifier. The classifier has a neural network. The sensor arrangement can be connected to a higher-level computer network via an interface. The control and evaluation unit is configured for an extension function. For this purpose, the sensor arrangement transmits selected sensor data, which is to be additionally processed by the classifier, to the computer network via an extension function. There, a classifier is trained based on the sensor data, and after completion of the training, the sensor arrangement receives classifier data back for modification of the classifier, such as parameters, program sections, or even the entire trained classifier.This means the sensor arrangement is also equipped for the classification of the selected sensor data.

[0005] US 2018 / 0053091 A1 describes a method for selecting and implementing a neural network for an embedded system. The method includes selecting a neural network from a library of neural networks based on parameters of the embedded system, where the parameters restrict the selection of the neural network. The method also includes training the neural network using a dataset. Furthermore, the method includes compressing the neural network for implementation on the embedded system.

[0006] Another example is the analysis of video data within the framework of semantic segmentation. According to the current state of technology, functions that utilize neural networks are indispensable in this area.

[0007] Neural networks not only require significant computing power, but are also generally highly complex and therefore extremely difficult to understand in their functioning. This fact also presents a major challenge in the area of ​​safety assurance, which is necessary for safety-relevant functions in the automotive context. Securing a system that is not understood is impossible.

[0008] For conventional algorithms, methods exist to mathematically prove their correctness. However, these methods are exhausted when using neural networks, which is why other approaches are required.

[0009] AI models based on neural networks exhibit different properties depending on the training data and the model structure. Training data can include, for example, data from training drives or synthetic training data. The properties of AI models are particularly evident in their functional and non-functional quality characteristics. Many of these properties are safety-relevant, such as the contextual dependence of functional performance, the stability of the function, etc. It is virtually impossible to develop an AI model that possesses all safety-relevant capabilities sufficiently to meet an acceptable safety concept. Furthermore, changing requirements that arise during driving operations necessitate varying degrees of certain capabilities in dynamic fluctuations.

[0010] Current approaches to using artificial intelligence in motor vehicles are limited to the use of a single AI module or an ensemble of AI modules and their optimization. An AI module, in this context, refers to a software component in which an AI model is implemented. Redundancy concepts are classically addressed, as in aviation, through multiple executions in conjunction with monitoring systems. Identical software components are executed synchronously two or three times, and their results are compared and utilized accordingly.

[0011] However, these redundancy concepts were not developed for AI modules, but for software components whose behavior is relatively easy to verify. AI modules, in their functionality, cover a gray area between "right" and "wrong," which is ignored by these redundancy concepts.

[0012] DE 10 2016 000 493 A1 describes a method for operating a vehicle system designed for fully automatic control of a motor vehicle in various driving situation classes. The method uses a computational structure comprising several evaluation units to determine control data for fully automatic vehicle control from environmental data describing the vehicle's surroundings and ego data describing the vehicle's state. Each evaluation unit determines output data from the output data of at least one other evaluation unit or from driving situation data. At least some of the evaluation units are implemented as neural networks, at least partially, on a software basis.At least some of the evaluation units, designed as a neural network, are dynamically generated during runtime from a configuration object configurable via configuration parameter sets. To do this, a current driving situation class is determined from several predefined driving situation classes using at least some of the driving situation parameters. Each driving situation class is assigned at least one evaluation function. Configuration parameter sets assigned to the evaluation functions of the current driving situation class are then retrieved from a database. Finally, any evaluation units that do not yet exist and execute the evaluation function are created by configuring configuration objects with the retrieved configuration parameter sets.

[0013] One objective of the invention is to demonstrate an improved concept for configuring a control system for an at least partially autonomous motor vehicle.

[0014] This problem is solved by a method having the features of claim 1, by a computer program with instructions according to claim 6, and by a device having the features of claim 7. Preferred embodiments of the invention are the subject of the dependent claims.

[0015] According to a first aspect of the invention, a method for configuring a control system for an at least partially autonomous motor vehicle comprises the following steps: Selecting two or more AI modules from a library of AI modules during driving operation based on selection criteria, wherein the selection criteria relate to functional properties of the AI ​​modules concerning object recognition by the AI ​​modules and non-functional properties of the AI ​​modules concerning the required computing capacity of the AI ​​modules; and initiating an execution of the two or more AI modules by the control system, wherein, during the execution of the AI ​​modules, properties of the two or more AI modules are used for an aggregation of the results of the two or more AI modules for a current context.

[0016] According to another aspect of the invention, a computer program comprises instructions which, when executed by a computer, cause the computer to perform the following steps for configuring a control system for an at least partially autonomous motor vehicle: Selecting two or more AI modules from a library of AI modules during driving operation based on selection criteria, wherein the selection criteria relate to functional properties of the AI ​​modules concerning object recognition by the AI ​​modules and non-functional properties of the AI ​​modules concerning the required computing capacity of the AI ​​modules; and initiating an execution of the two or more AI modules by the control system, wherein, during the execution of the AI ​​modules, properties of the two or more AI modules are used for an aggregation of the results of the two or more AI modules for a current context.

[0017] The term "computer" is to be understood broadly. In particular, it also includes control units and other processor-based data processing devices.

[0018] The computer program can, for example, be made available for electronic retrieval or be stored on a computer-readable storage medium.

[0019] According to another aspect of the invention, a device for configuring a control system for an at least partially autonomous motor vehicle comprises: A selection unit for selecting two or more AI modules from a library of AI modules during driving operation based on selection criteria, wherein the selection criteria relate to functional properties of the AI ​​modules concerning object recognition by the AI ​​modules and non-functional properties of the AI ​​modules concerning the required computing capacity of the AI ​​modules; and an execution control for initiating an execution of the two or more AI modules by the control system, wherein, during the execution of the AI ​​modules, properties of the two or more AI modules are used for an aggregation of the results of the two or more AI modules for a current context.

[0020] Previous approaches are limited to using a single AI model for a specific function, which is applied unchanged to all driving scenarios. In contrast, the approach according to the invention does not focus on developing a single function that works well and safely in all situations, but rather on combining a set of effective functions. The various AI modules are dynamically combined and executed redundantly on the vehicle side. Any available AI module can be integrated into the vehicle system, regardless of whether it was developed internally or externally. An inherent feature of this approach is that it always delivers results at least as good as those of the best function included in the selection. However, the results achieved through dynamic selection are generally better than those that each function executed by the control system would deliver individually.This results in an increase in overall performance compared to the performance of the individual modules. The solution according to the invention is characterized in particular by its high system availability, its security, and its adaptability.

[0021] The AI ​​modules are stored in a library within the vehicle, which can contain hundreds or more modules. Based on selection criteria, several AI modules are chosen from this library for active operation. The computing power available in future vehicles will be sufficient to operate, for example, three to four AI models in parallel. To ensure the reliability of relevant functions, these modules are redundantly configured to compensate for any failures or errors; that is, several AI modules are available that perform the same function. This redundancy allows for synchronous quality control of the active AI modules and enables a priori known loss of efficiency or effectiveness in the event of a failure of one or more AI modules.

[0022] When combining AI modules in a vehicle, several objectives are pursued, which are either functional or non-functional in nature. In particular, it is not necessarily required to choose the result of one of the redundantly available AI modules. Rather, combining the individual results to improve both functional and non-functional performance is also possible.

[0023] The selection criteria concern properties of the AI ​​modules that influence which of the available AI modules should be used for a given situation. These properties include functional characteristics of the AI ​​modules regarding object recognition, such as the detection of specific objects, rare objects, hidden objects, and the consistency of the recognition. The properties also include non-functional characteristics of the AI ​​modules concerning the required computing power. By considering these selection criteria, optimal system performance can be achieved for every situation.

[0024] According to the invention, properties of two or more AI modules are used to aggregate the results of these modules for a given context. This aggregation can be achieved, for example, by computation, substitution, or object-wise or class-wise selection of the results from the two or more AI modules. Utilizing the properties of the AI ​​modules or the underlying AI models for aggregation allows for the generation of optimal output for the current context. This involves, for example, using properties that contain information about the performance of the individual modules with respect to specific segmentation classes or contexts. For instance, an AI model might exhibit higher performance for segmenting pedestrians in an urban environment at night. This could be reflected in a higher confidence value within the module properties.

[0025] According to one aspect of the invention, the AI ​​modules are evaluated by a backend. For this purpose, the AI ​​module or the underlying AI model is checked by the backend for specific properties and evaluated meaningfully, taking into account the semantic context. Semantic contexts include, for example, the driving situation, the environment, the weather, the time of day, etc. The evaluation results are stored in the module properties and can be used by the vehicle for module selection. The backend can also transmit new or modified AI modules to the vehicle.

[0026] According to one aspect of the invention, when a newly selected AI module is started, the AI ​​module being replaced remains active until the newly selected AI module has fully started. While an AI module is starting, it is not yet fully operational but already requires computing resources. If one of the active AI modules is then replaced by a previously inactive one, the AI ​​module being replaced remains active until the new AI module has fully started up. Although this requires more computing resources in the short term, in the medium term this method enables a robust and secure system.

[0027] According to one aspect of the invention, three or four redundant AI modules are selected for execution. The parallel operation of three to four AI modules optimally utilizes the computing capacities that will be available in motor vehicles in the future.

[0028] A method or device according to the invention is particularly advantageous when used in a motor vehicle.

[0029] Further features of the present invention will become apparent from the following description and the attached claims in conjunction with the figures. Fig. 1 schematically shows a method for configuring a control system for an at least partially autonomous motor vehicle; Fig. 2 shows a first embodiment of a device for configuring a control system for an at least partially autonomous motor vehicle; Fig. 3 shows a second embodiment of a device for configuring a control system for an at least partially autonomous motor vehicle; Fig. 4 schematically represents a motor vehicle in which a solution according to the invention is implemented; Fig. 5 schematically shows a system diagram of a solution according to the invention.

[0030] To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments and that the described features can also be combined or modified without leaving the scope of protection of the invention as defined in the appended claims.

[0031] Fig. 1Figure 10 schematically illustrates a procedure for configuring a control system for a partially autonomous vehicle. In a first step, data on the driving situation is collected. This can include, for example, environmental data from sensors installed in the vehicle or vehicle operating parameters provided by control units. Selection criteria are then determined from the available data. These criteria relate to functional properties of the AI ​​modules concerning object recognition and non-functional properties such as the required computing capacity. Additional constraints, such as available computing capacity, can also be considered. Based on the selection criteria, two or more AI modules are selected from a library of AI modules during driving.Preferably, three or four redundant AI modules are selected. Finally, the control system initiates the execution of the selected two or more AI modules. During execution, the AI ​​modules' properties are used to aggregate their results for a given context. This aggregation can be achieved, for example, by computation, substitution, or object-wise or class-wise selection of the AI ​​module results. Preferably, when a newly selected AI module is started, the AI ​​module to be replaced remains active until the newly selected AI module has finished running.

[0032] Fig. 2Figure 1 shows a simplified schematic representation of a first embodiment of a device 20 for configuring a control system for an at least partially autonomous motor vehicle. The device 20 has an input 21 through which data on the driving situation can be received. This can be, for example, environmental data from sensors installed in the motor vehicle or operating parameters of the motor vehicle provided by control units. An evaluation unit 22 then determines selection criteria from the available data. The selection criteria relate to functional properties of the AI ​​modules concerning object recognition by the AI ​​modules and non-functional properties of the AI ​​modules concerning the required computing capacity of the AI ​​modules. Additional boundary conditions, such as available computing capacity, can be taken into account if necessary.Based on the selection criteria, a selection unit 23 selects two or more AI modules from a library of AI modules during operation. Preferably, three or four redundant AI modules are selected by the selection unit 23. The library can, for example, be stored in a memory 26 of the device 20. An execution controller 24 initiates the execution of the two or more AI modules by the control system. The control system is made available for further use via an output 27 of the device 20. For example, the used AI modules can be directly output to a processor. Alternatively, only information about which AI modules should be loaded can be output to a processor. During the execution of the AI ​​modules, properties of the AI ​​modules are used to aggregate the results of the AI ​​modules for a current context. The aggregation can, for example,This can be achieved by calculating, replacing, or selecting the results of the AI ​​modules on an object-by-object or class-by-class basis. Preferably, the execution control 24 ensures that when a newly selected AI module is started, the AI ​​module to be replaced remains active until the newly selected AI module has fully started.

[0033] The evaluation unit 22, the selection unit 23, and the execution controller 24 can be controlled by a control unit 25. Settings of the evaluation unit 22, the selection unit 23, the execution controller 24, or the control unit 25 can be changed via a user interface 28. The data generated in the device 20 can be stored in the memory 26 as needed, for example, for later evaluation or for use by the components of the device 20. The evaluation unit 22, the selection unit 23, the execution controller 24, and the control unit 25 can be implemented as dedicated hardware, for example, as integrated circuits. Of course, they can also be partially or fully combined or implemented as software running on a suitable processor, such as a GPU or a CPU.Input 21 and output 27 can be implemented as separate interfaces or as a combined bidirectional interface.

[0034] Fig. 3Figure 3 shows a simplified schematic representation of a second embodiment of a device 30 for configuring a control system for an at least partially autonomous motor vehicle. The device 30 comprises a processor 32 and a memory 31. For example, the device 30 is a computer or a control unit. Instructions are stored in the memory 31 which, when executed by the processor 32, cause the device 30 to perform the steps according to one of the described methods. The instructions stored in the memory 31 thus embody a program executable by the processor 32, which implements the method according to the invention. The device 30 has an input 33 for receiving information, for example, environmental data or operating parameters of the motor vehicle. Data generated by the processor 32 is provided via an output 34. Furthermore, it can be stored in the memory 31.Input 33 and output 34 can be combined into a bidirectional interface.

[0035] The processor 32 can comprise one or more processor units, such as microprocessors, digital signal processors, or combinations thereof.

[0036] The memory elements 26, 31 of the described embodiments can have both volatile and non-volatile memory areas and can include a wide variety of storage devices and storage media, for example hard disks, optical storage media or semiconductor memory.

[0037] Fig. 4 Figure 1 schematically represents a motor vehicle 40 in which a solution according to the invention is implemented. The motor vehicle 40 has a control system 41 for automated or highly automated driving operation, which is configured by a device 20. Fig. 4The device 20 is a standalone component, but it can also be integrated into the control system 41. To select AI modules from a library of AI modules, the device 20 uses data about the driving situation. This data can include, for example, environmental data from sensors 42 installed in the vehicle 40 or operating parameters of the vehicle 40 provided by control units 43. Another component of the vehicle 40 is a data transmission unit 44, which, among other things, allows a connection to be established to a backend, e.g., to obtain additional or modified AI modules. A memory 45 is provided for storing the library of AI modules or other data. Data exchange between the various components of the vehicle 40 takes place via a network 46.

[0038] Fig. 5Figure 1 schematically shows a system diagram of a solution according to the invention. The system comprises a motor vehicle 40, an external source 50 of AI modules NN i, and a backend 60. A library B of AI modules NN i is stored in a memory 45 of the motor vehicle 40. These modules may have been developed internally by a manufacturer of the motor vehicle 40 or may originate from the external source 50. In this example, the AI ​​module NN 1 is provided by the external source 50. Regardless of their origin, the AI ​​modules NN i are evaluated by the backend 60. The backend 60 considers functional properties of the AI ​​modules NN i relating to object recognition and non-functional properties of the AI ​​modules NN i relating to the required computing capacity of the AI ​​modules. Examples of functional properties relating to object recognition are, for example...The recognition of specific objects, the recognition of rare objects, the recognition of hidden objects, the consistency of the recognition, etc. The resulting evaluations of the AI ​​modules NN i are stored in library B as module properties ME. Based on selection criteria and a comparison with the module properties ME, various AI modules are dynamically combined on the vehicle side and executed redundantly by a control system 41 of the vehicle 40. In the example, these are the two AI modules NN 1 and NN 2. Reference symbol list

[0039] 10 Acquisition of data on the driving situation 11 Determination of selection criteria from the available data 12 Selection of two or more AI modules 13 Initiation of execution of the two or more AI modules 20 Device 21 Input 22 Evaluation unit 23 Selection unit 24 Execution control 25 Control unit 26 Memory 27 Output 28 User interface 30 Device 31 Memory 32 Processor 33 Input 34 Output 40 Motor vehicle 41 Control system 42 Environmental sensors 43 Control unit 44 Data transmission unit 45 Memory 46 Network 50 External source 60 Backend Library ME Module properties NN i AI module

Claims

1. Method for configuring a control system (41) for an at least partially autonomous motor vehicle (40), which method comprises the steps of: - selecting (12) two or more AI modules (NNi) from a library (B) of AI modules during driving operation, on the basis of selection criteria, wherein the selection criteria relate to at least functional properties of the AI modules (NNi) relating to object recognition by the Al modules (NNi); and - initiating (13) execution of the two or more AI modules (NNi) by the control system (41); characterized in that the selection criteria also relate to non-functional properties of the AI modules (NNi) relating to a required computing capacity of the AI modules (NNi), and in that, during the execution of the Al modules (NNi), properties of the two or more Al modules (NNi) are used for aggregating the results of the two or more AI modules (NNi) for a current context.

2. Method according to claim 1, wherein the AI modules (NNi) are evaluated by a backend (60).

3. Method according to claim 1 or 2, wherein the aggregation is performed by calculating, replacing or selecting, by object or by class, the results of the two or more AI modules (NNi).

4. Method according to any of the preceding claims, wherein, when a newly selected Al module (NNi) is started, an Al module (NNi) to be replaced remains active until the newly selected Al module (NNi) has been fully started.

5. Method according to any of the preceding claims, wherein three or four redundant Al modules (NNi) are selected (12) for execution.

6. Computer program having instructions which, when executed by a computer, cause the computer to execute the steps of a method according to any of claims 1 to 5 for configuring a control system (41) for an at least partially autonomous motor vehicle (40).

7. Device (20) for configuring a control system (41) for an at least partially autonomous motor vehicle (40), which device comprises: - a selection unit (23) for selecting (12) two or more AI modules (NNi) from a library (B) of AI modules during driving operation, on the basis of selection criteria, wherein the selection criteria relate to at least functional properties of the AI modules (NNi) relating to object recognition by the AI modules (NNi); and - an execution controller (24) for initiating (13) execution of the two or more AI modules (NNi) by the control system (41); characterized in that the selection criteria also relate to non-functional properties of the AI modules (NNi) relating to a required computing capacity of the Al modules (NNi), and in that, during the execution of the Al modules (NNi), properties of the two or more AI modules (NNi) are used for aggregating the results of the two or more AI modules (NNi) for a current context.