Method and device for evaluating a quality of an automated function of a motor vehicle

Abstract

The invention relates to a method for evaluating a quality of an automated function of a motor vehicle, wherein the method comprises providing at least one self-analysis function as a software module for the automated function, determining a predefined variable based on input data and / or output data of the automated function by means of the self-analysis function, and transmitting the predefined variable from the motor vehicle to a backend.

Description

[0001] METHOD AND DEVICE FOR EVALUATING THE QUALITY OF AN AUTOMATED FUNCTION OF A MOTOR VEHICLE

[0002] The present disclosure relates to a method for evaluating the quality of an automated function of a motor vehicle. Additionally or alternatively, a data processing device or a data processing system is provided which is designed to at least partially carry out the method. Additionally or alternatively, an (optionally automated) motor vehicle with part of the data processing device is provided. Additionally or alternatively, a computer program is provided which comprises instructions which, when the program is executed by a computer, cause the computer to at least partially carry out the method. Additionally or alternatively, a computer-readable medium is provided which comprises instructions which, when the instructions are executed by a computer, cause the computer to at least partially carry out the method.

[0003] Via mobile data connections, data or information from (customer) vehicles can be transmitted to a backend or cloud for data collection and analysis. Traditionally, existing data interfaces are used for this purpose, and the data is sent via stream or buffer. For cost reasons, only a few signals can be sent at low frequencies.

[0004] Typical signals provided upstream include availability information about customer functions or automated functions (or automated systems), or information about whether and at which position the customer has enabled (or disabled) such a function, or where a problem has occurred with such a function. This information allows for analyses of existing functions, limited to situations in which they are active or activatable according to their operational design domain (ODD).

[0005] The ODD can be understood as a description or definition of specific operating ranges in which an automated function or system is intended to function properly, including, but not limited to, road types, speed ranges, environmental conditions (weather, time of day / night, etc.), and / or other operating range restrictions. In other words, the ODD can be understood as the specific conditions under which a particular driving automation system or function thereof is intended to function, including, but not limited to, driving modes. This can include a variety of restrictions, such as geographical, traffic, speed, and / or road-related restrictions.

[0006] However, novel functions or functions outside their ODD cannot be analyzed because the ability to analyze is linked to the activation or activatability of the function.

[0007] Against the background of this prior art, the object of the present disclosure is to provide a device and a method which are each suitable for enriching the prior art.

[0008] The problem is solved by the features of the independent claim. The subordinate claims and subclaims contain possible further developments of the disclosure.

[0009] The task is then solved by a method for evaluating the quality of an automated function of a motor vehicle.

[0010] The term quality can be understood as an objective quantity or an objective measure of the functionality or quality of an automated function of the motor vehicle, in particular of a sub-system of such a function (e.g. a fusion) or of a software component thereof.

[0011] The method can be a computer-implemented method, i.e. one, several or all steps of the method can be carried out by a computer or a data processing device. The automated function can be understood as a function or a (software and / or hardware) component which performs one or more predetermined tasks in the motor vehicle automatically or without human intervention. The automated function can be part of a driver assistance system or represent such a system. Examples of an automated function include object detection, trajectory planning and / or direct and / or indirect intervention or direct and / or indirect control of lateral and / or longitudinal guidance of the motor vehicle.

[0012] The method comprises providing at least one self-analysis function as a software component for the automated function.

[0013] The method comprises determining a predetermined value based on input data and / or output data of the automated function by means of the self-analysis function.

[0014] In other words, an algorithm, the so-called self-analysis function, determines a quantity or piece of information that is determined by a configuration of the algorithm. Thus, the value of the quantity or piece of information is not predetermined or predefined, but rather the quantity to be determined. The respective algorithm can specify which signal or data stream is to be tapped in the motor vehicle and how the tapped signal is to be analyzed to obtain the (thus predetermined) quantity.

[0015] The method comprises transmitting the (determined) predetermined value from the motor vehicle to a backend.

[0016] The transmission or transfer of the quantity can be wireless, e.g., via the internet and / or the mobile network. The transmission of the quantity and the execution of the method can be continuous, particularly cyclically or continuously. The transmission can then also be referred to as streaming (so-called upstream).

[0017] In other words, customer functions or automated functions are usually composed of many components, such as object detection, road detection, a trajectory planner, etc. It is proposed to supplement existing components or those already present in the vehicle (e.g., a recognition function such as object detection) with a self-analysis function or "self-assessment" functionalities. Their task is to determine characteristic values ​​(as part of determining the so-called predetermined value) that allow the quality of the component to be assessed.

[0018] The method can be used to analyze functional enhancements and risks based on the self-assessment or self-analysis of on-board algorithms. This makes it possible to evaluate and ultimately implement new automated functions and / or ODD extensions of existing automated functions.

[0019] New functions can be built on existing building blocks if signals or information provided for and / or by automated functions already installed in the vehicle are used in a novel way (e.g., a specific new reaction to pedestrians detected by existing automated functions who display a particular behavior pattern). Since the new automated function does not exist in the vehicle, it cannot be evaluated using conventional means across the customer fleet. However, the procedure described above can be used to evaluate the extent to which the building blocks underlying the new automated function provide signals or information of a quality sufficient for the new automated function (e.g., is the pedestrian detection good enough to be able to recognize the particular behavior pattern?). This can provide an understanding of whether these signals, for example,are available across the board or with certain restrictions (depending on situations, location, etc.) in the required quality. As mentioned above, ODD extensions of existing automated functions are also possible with the method. ODDs are deliberately restricted in their functional design in the expectation that an automated function cannot be offered with sufficient quality in certain situations (e.g., an automated function that controls for objects is not offered on small streets, depending on the road class, because such road classes are likely to contain particularly difficult-to-detect objects, and the installed sensors cannot nominally achieve this).If one now wants to evaluate whether a function can be operated outside of its nominal ODD, or which measures would be necessary to extend its ODD, an evaluation of the behavior of the individual function modules in the ODD areas to be examined is necessary and is possible using the procedure described above by a suitable definition of the predetermined quantities or data or information to be determined by the self-analysis function.

[0020] The procedure described above is explained in detail below with reference to possible further training.

[0021] The predetermined value can be an uncertainty in an association of an object detected by a sensor of the motor vehicle with an object track, a frequency of a change of hypothesis in the case of an ambiguous detection of an object detected by a sensor of the motor vehicle, a frequency of a class change of an object detected by a sensor of the motor vehicle, and / or an a posteriori measure for a misdetection of an object detected by a sensor of the motor vehicle.

[0022] In other words, a measure of the uncertainty / ambiguity in the association of sensor objects to object tracks, a measure of the frequency of hypothesis changes in the case of ambiguous sensor detections, a measure of the frequency of ID changes of tagged entities, and / or an a posteriori measure for misdetections (e.g., how often were objects misclassified at a distance, but this was only apparent upon observation after approaching at close range) can be used.

[0023] The method may comprise providing a plurality of self-analysis functions each as a software module for the automated function, determining a predetermined value based on the input data and / or the output data of the automated function by means of the self-analysis functions, determining a data vector from the predetermined values ​​determined by means of the self-analysis functions, and transmitting the data vector from the motor vehicle to the backend.

[0024] The method may comprise determining a position of the motor vehicle at which the predetermined size is determined and transmitting the determined position together with the predetermined size from the motor vehicle to the backend.

[0025] The method may include creating a digital map in the backend based on the transmitted predetermined size and the position transmitted along with the predetermined size. The digital map may then include a description of, for example, expected functional limitations, risks, etc., at specific geolocations or at specific points, sections, and / or in specific areas of the digital map.

[0026] The method can comprise performing the method described above several times in order to obtain a plurality of predetermined variables or a plurality of data vectors, which are determined at the same position of the motor vehicle and / or in the same driving situation, in the backend. Furthermore, the method can comprise aggregating the plurality of predetermined variables or the plurality of data vectors in the backend, wherein the digital map can be created based on the aggregated plurality of predetermined variables or the plurality of data vectors. The digital map can further be created based on an availability of the automated function at the respective position of the motor vehicle and / or information about an ODD range of the automated function. The digital map can then contain information about the availability of the automated function at specific geolocations orat specific locations, sections and / or in specific areas of the digital map.

[0027] In other words, the above description can be summarized as follows, with reference to a specific embodiment that is described as non-limiting for the present disclosure: A plurality of such "self-assessment" functionalities can be operated, and a resulting data vector can be streamed, for example, via a permanent up channel to a data backend. There, a map can then be created that displays the data vectors, aggregated from many customer or developer fleet vehicles, together with function ODD ranges and information on customer function availability. This results in the following possibilities, among others: The correlation between the "self-assessment" vectors and customer function behavior can be evaluated. For example, increased ambiguities in object detection can be correlated with improper braking onto objects.Additionally or alternatively, risky locations and situations can be identified where the self-assessment vectors indicate poor or faulty behavior that may not yet have led to functional impairments. For example, frequent ID changes of tracked lanes can be observed in tight single-lane curves, but these never led to complaints in tests because, due to the single-lane nature, no error was made when re-establishing the lane. This has now identified a risk that could become relevant for tight multi-lane curves, which can be quantified and evaluated using fleet data. Additionally or alternatively, opportunity-laden locations and situations can be identified where the self-assessment vectors indicate sufficiently good behavior to operate existing functions outside their ODD or to provide new functions.As an example, consider a function that is based on good object detection but has never been released for small road types. The data may now show that the detection quality is indeed sufficient to extend the function to these small road types as well. Additionally or alternatively, logical conditions can be created from the linking of "self-assessment" vectors. This logic can be provided to the on-board system via a down-channel (i.e., from the backend to the vehicle) as part of specific data collection campaigns. From this, the on-board system creates trigger conditions that initiate an upload of certain (or all available) sensor data in a period before and / or after the trigger event. This data can then be analyzed in depth in the backend, for example to analyze the reasons for quality losses in detection functions in a specific context.An example of this could be a driving function that rarely exhibits errors in roundabouts. The self-assessment vectors then reveal that these cases correlate with anomalies during lane ID changes, increased uncertainty in object detection, and slow oncoming traffic. The combination of these three elements of the self-assessment vectors (ID changes, uncertainty, oncoming traffic speed) together with the location information "in a roundabout" is provided to the on-board system as a trigger condition. This allows data from the field with these specific anomalies to be collected and analyzed, and improvements to be derived in the form of software updates.

[0028] Furthermore, a data processing system comprising means for carrying out the computer-implemented method described above is provided.

[0029] The system may be a distributed system that may include a data processing device external (to the motor vehicle) (ie the backend), which is operated, for example, by the motor vehicle manufacturer, and / or a data processing device connected thereto (in particular wirelessly) and arranged in or on the motor vehicle.

[0030] The latter data processing device can therefore be part of a motor vehicle. The data processing device of the motor vehicle can be, for example, a data logger and / or an electronic control unit (ECU). The data processing device of the motor vehicle can be an intelligent processor-controlled unit that can communicate with other modules, for example, via a central gateway (CGW), and which can optionally form the vehicle's on-board network via fieldbuses such as the CAN bus, LIN bus, MOST bus, and FlexRay, or via automotive Ethernet, e.g., together with telematics control units. The data processing device can be connected to a sensor system, e.g., via the aforementioned on-board network.

[0031] It is conceivable that the data processing device comprises a storage unit in which collected data and / or the determined quantity(s) are temporarily stored before being sent to the external data processing device. It is conceivable that the data processing device sends the collected data to the data processing device operated by the motor vehicle manufacturer, in particular by streaming (optionally without intermediate storage).

[0032] What has been described above with reference to the procedure also applies analogously to the data processing system and vice versa.

[0033] Furthermore, a motor vehicle comprising a data processing device can be provided, wherein the data processing device is designed to carry out the method for evaluating the quality of the automated function of the motor vehicle, the method comprising: determining a predetermined value based on input data and / or output data of the automated function by means of a self-analysis function, and transmitting the predetermined value from the motor vehicle to the backend.

[0034] The motor vehicle may be a passenger car, in particular an automobile. The motor vehicle may be an automated motor vehicle, which may be configured to at least partially and / or at least temporarily assume longitudinal and / or lateral guidance during automated driving of the motor vehicle. Automated driving may be implemented in such a way that the motor vehicle's movement is (largely) autonomous.

[0035] The motor vehicle may be a motor vehicle with autonomy level 0, i.e. the driver assumes the dynamic driving task, even if supporting systems (e.g. ABS or ESP) are present.

[0036] The motor vehicle may be a Level 1 autonomy motor vehicle, i.e. it may have certain driver assistance systems that support the driver in operating the vehicle, such as adaptive cruise control (ACC).

[0037] The motor vehicle may be a motor vehicle of autonomy level 2, i.e. be so partially automated that functions such as automatic parking, lane keeping or lateral guidance, general longitudinal guidance, acceleration and / or braking are taken over by driver assistance systems.

[0038] The motor vehicle can be a Level 3 autonomy vehicle, meaning it is so conditionally automated that the driver does not need to continuously monitor the vehicle system. The vehicle performs functions such as activating the turn signal, changing lanes, and / or keeping in lane independently. The driver can attend to other tasks but will be prompted by the system to take over control within a pre-warning period if necessary.

[0039] The motor vehicle can be a Level 4 autonomy vehicle, meaning it is so highly automated that the vehicle system permanently assumes control of the vehicle. If the system can no longer handle the driving tasks, the driver can be requested to take over.

[0040] The motor vehicle can be a Level 5 autonomy vehicle, meaning it is so fully automated that the driver is not required to perform the driving task. Apart from setting the destination and starting the system, no human intervention is required. The above description with regard to the method and the data processing system applies analogously to the motor vehicle, and vice versa.

[0041] Furthermore, a computer program can be provided. The computer program is characterized in that it comprises instructions which, when executed by a computer, cause the computer to at least partially execute the method described above.

[0042] A program code of the computer program can be in any code, in particular in a code that is suitable for controlling motor vehicles, e.g. comprising LUA scripts.

[0043] Furthermore, a computer-readable medium, in particular a computer-readable storage medium, is provided. The computer-readable medium is characterized in that it comprises instructions which, when the program is executed by a computer, cause the computer to at least partially execute the method described above.

[0044] This means that a computer-readable medium can be provided that contains a computer program as defined above. The computer-readable medium can be any digital data storage device, such as a USB flash drive, a hard disk, a CD-ROM, an SD card, or an SSD card.

[0045] The computer program does not necessarily have to be stored on such a computer-readable storage medium in order to be made available to the motor vehicle, but can also be obtained via the Internet or otherwise externally.

[0046] The above description with reference to the method, the data processing system, and the motor vehicle also applies analogously to the computer program and the computer-readable medium, and vice versa. An embodiment is described below with reference to Figures 1 and 2.

[0047] Fig. 1 shows schematically a data processing system comprising means for carrying out a method for evaluating a quality of an automated function of a motor vehicle, and

[0048] Fig. 2 shows a schematic flow diagram of the process.

[0049] The system 1 shown in Figure 1 has a backend 2 and a motor vehicle 3 connected to the backend via a wireless data connection 4. The motor vehicle 3 has a data processing device 31 and a module or part 32 connected thereto (e.g., a driver assistance system) which is configured to perform an automated function (e.g., object recognition based on sensor data from a sensor system (not shown) of the motor vehicle 3). The system 1 is configured to execute the method for evaluating the quality of the automated function, also described below with reference to Figure 2.

[0050] In a first step S1 of the method, several self-analysis functions are provided for the automated function, each as a software module or collectively as a (single) software module. For this purpose, the self-analysis functions can be transmitted, for example, via the wireless data connection 4 from the backend 2 to the motor vehicle 3, more precisely to the data processing device 31.

[0051] In a second step S2 of the method, at least one predetermined variable is determined based on input data and / or output data of the automated function using the self-analysis functions executed by the data processing device 31. The input data can be sensor data, e.g., image data. The output data can be, for example, a result of an analysis of the sensor data performed by the automated function.The predetermined variable can be an uncertainty in the association of an object detected by a sensor of the motor vehicle with an object track, a frequency of a hypothesis change in the event of an ambiguous detection of an object detected by a sensor of the motor vehicle, a frequency of a class change of an object detected by a sensor of the motor vehicle, and / or an a posteriori measure for a false detection of an object detected by a sensor of the motor vehicle. Furthermore, a position of the motor vehicle at which the predetermined variables are determined is determined. In addition, an availability of the automated function at the respective position of the motor vehicle 3, i.e., at the time at which the predetermined variables are determined, and information about an ODD range of the automated function are determined.

[0052] In a third step S3 of the method, a data vector comprising the predetermined variables determined by means of the self-analysis functions, the determined position of the motor vehicle, the availability of the automated function at the time of determining the position and the predetermined variables, and the information about the ODD range of the automated function is determined by the data processing device 31.

[0053] In a fourth step S4 of the method, the data vector is transmitted from the motor vehicle 3 to the backend 2.

[0054] The steps S1 - S4 described above are carried out repeatedly or several times by several motor vehicles of a motor vehicle fleet to which the motor vehicle 3 belongs, so that the predetermined variables and thus the data vector based thereon are determined several times at the same position (at different times) and / or in the same driving situation (e.g. driving through a roundabout) and transmitted from the respective motor vehicle to the backend 2.

[0055] In a fifth step S5 of the method, the multiple data vectors are aggregated in the backend 2. This can be done, for example, by calculating an average of the determined variables contained in the data vectors. In a sixth step S6 of the method, an existing digital map is created or supplemented based on the multiple aggregated data vectors.

[0056] In a seventh step S7 of the method, the quality of the automated function is evaluated based on predetermined or predefined criteria using the digital map and the automated function is adapted, e.g., by extending its ODD range and / or determining a further automated function based on a result of the analysis.

[0057] The adapted and / or further automated function can be implemented by means of a software update via the wireless data connection 4 in an eighth step S8 of the method in the motor vehicle 3, in particular the entire motor vehicle fleet to which the motor vehicle 3 belongs.

[0058] List of reference symbols

[0059] 1 Data processing system 2 Backend

[0060] 3 Motor vehicle

[0061] 31 Data processing device

[0062] 32 module with automated function

[0063] 4 wireless data connection

[0064] S1 - S8 process steps

Claims

Patent claims Method for evaluating the quality of an automated function of a motor vehicle (3), wherein the method comprises: Providing at least one self-analysis function as a software component for the automated function, - Determining a predetermined value based on input and / or output data of the automated function using the self-analysis function, and - Transmitting the predetermined size from the motor vehicle (3) to a backend (2). Method according to claim 1, wherein the predetermined size is: - an uncertainty in associating an object detected by a sensor of the motor vehicle (3) with an object track, - a frequency of hypothesis change in the case of an ambiguous detection of an object detected by means of a sensor of the motor vehicle (3), - a frequency of a class change of an object detected by means of a sensor of the motor vehicle (3), and / or - an a posteriori measure of a false detection of an object detected by a sensor of the motor vehicle (3). Method according to claim 1 or 2, wherein the method: - Providing a wide range of self-analysis functions as a software component for automated operation, - Determining a predetermined value based on the input data and / or the output data of the automated function using the self-analysis functions, Determining a data vector from the predetermined quantities determined using the self-analysis functions, and - Transmitting the data vector from the motor vehicle (3) to the backend (2).

4. A method according to any one of claims 1 to 3, wherein the method comprises: - Determining a position of the motor vehicle (3) at which the predetermined quantity is determined, and - Transmitting the determined position together with the predetermined size from the motor vehicle (3) to the backend (2).

5. Method according to any one of claims 1 to 4, wherein the method comprises creating a digital map based on the transmitted predetermined size and the position transmitted together with the predetermined size in the backend.

6. The method of claim 5, wherein the method comprises: - repeated execution of the procedure according to one of the claims 1 to 4, to obtain several predetermined quantities or several data vectors, which are determined at the same position of the motor vehicle (3) and / or in the same driving situation, in the backend (2), and - Aggregating the multiple predetermined sizes or the multiple data vectors in the backend (2), - where the creation of the digital map is based on the aggregated multiple predetermined sizes or multiple data vectors.

7. Method according to claim 5 or 6, wherein the creation of the digital map is further based on the availability of the automated function at the respective position of the motor vehicle (3) and / or information about an ODD area of ​​the automated function.

8. Data processing system (1) comprising means for carrying out the method according to any one of claims 1 to 7.

9. Computer program comprising instructions that, when executed by a computer, cause the computer to execute the method according to any one of claims 1 to 7.

10. Computer-readable medium comprising instructions that, when executed by a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images