Computer-implemented method for automatically determining an operational design domain (ODD) coverage metric of an ai model applied in a safety critical system

Abstract

A computer-implemented method for automatically determining an Operational Design Domain (ODD) coverage metric for an AI model applied in a safety critical system wherein the Operational Design Domain specifies different ODD aspects concerning environmental objects and attributes the AI model must be able to identify, comprising: for each ODD aspect of a selection of ODD aspects, - obtaining (10) at least one concept (C1,..., Cn) comprising essential features of the ODD aspect via a user interface, - creating (11) a collection of images depicting the essential features of the concept, - determining (12) a score for each concept with respect to a target class (kt) learned by the AI model (Mc) by applying a concept-based explanation method to the AI model (Mc) for each class of the AI model (Mc), which is relevant for the concept (C1,..., Cn), and for each concept, and - outputting (13) the ODD coverage metric represented by the determined scores for each ODD aspect, via the user interface and an apparatus configured to perform the method.

Description

[0001] Description

[0002] Computer-implemented method for automatically determining an Operational Design Domain (ODD) coverage metric of an Al model applied in a safety critical system

[0003] Technical Field

[0004] The application concerns a computer-implemented method for automatically determining an Operational Design Domain (ODD) coverage metric for an Al model applied in a safety critical system wherein the Operational Design Domain specifies different ODD aspects concerning environmental objects and attributes the Al model must be able to identify, as well as an apparatus configured to perform the method and a respective computer program product.

[0005] Background Art

[0006] In Artificial Intelligence (Al) models the decision-making process is usually hidden from the user, and the Al models are treated as black boxes. However, in cases where Al models are used in systems where safety is paramount, this approach is not suitable.

[0007] For instance, to ensure more trust and safety for Al models applied in an autonomous vehicle (AV) constraints are defined which describe under which conditions the Al model is expected to function as intended. These constraints are described by an Operational Design Domain (ODD) which specifies what static or dynamic environmental parameters the Autonomous Vehicle must be able to manage. They include different types of traffic participants, weather conditions, infrastructure, location, time of day and many more that may impact the driving situation. Even if the Al system is demonstrated to possess adequate precision, it must also be assessed and demonstrated that the entire scope of the ODD has been intrinsically encompassed by the Al model.

[0008] Nowadays, certification of Al systems used in safety critical systems is mainly done by using machine learning approaches that are self-explaining, e.g., tree algorithms, or by rule-based models. Due to the complex nature of real-life problems, however, these approaches are hard to scale with respect to accuracy and at the same time completeness and explainability. The resulting rules quickly become overwhelming making it extremely hard to prove the ODD coverage. Another known way of assessing the ODD is to define a set of scenarios and simulate them to see the behavior of the autonomous vehicle. Later, the test results are used to evaluate how capable an autonomous vehicle is with respect to its ODD. In this approach, the enormous number of scenarios poses a challenge. Additionally, the definition of these scenarios is a manual process undertaken by the developers.

[0009] Concept-based explanation methods are known that aim to generate most prominent concepts automatically, e.g., by Ghorbani et al. 2019, published at https: / / arxiv.org / pdf / 1902.03129.pdf. In this approach, instead of assigning importance to individual features or pixels, the output of the method reveals the important concepts that lead to the prediction of the Al system. As an example, the resulting important concepts for detecting police vans are a wheel and a police logo. Seeing important concepts for a class will certainly be valuable, but these concepts are not correlated with conditions proposed by the ODD document.

[0010] Such prior art faces certain restrictions and drawbacks. For instance, the number of rules and scenarios which have to be defined and checked to evaluate the compliance of the Al model with respect to the ODD constraints become overwhelming making it extremely hard to prove the ODD coverage. A quantification of the coverage of the ODD constrains by the Al model is not available.

[0011] Summary of Invention

[0012] The technical object of the present application is to provide a quantifiable measure on deployment readiness of an Al model, which covers the constraints defined by the Operational Design Domain.

[0013] The basic idea concerns a method and an apparatus that calculates an ODD coverage metric for an Al model by utilizing concept learning in combination with human oversight. The intention is to create high-level human-interpretable concepts that are essential for an ODD aspect and make use of concept-based explanation methods that can generate scores for each concept and utilize these scores as an ODD coverage metric.

[0014] This object is solved by the features of the independent claims. Advantageous embodiments are provided in the dependent claims. A first aspect concerns a computer-implemented method for automatically determining an Operational Design Domain (ODD) coverage metric for an Al model, especially when applied in a safety critical system, wherein the Operational Design Domain specifies different ODD aspects concerning environmental objects and attributes the Al model must be able to identify, comprising: for each ODD aspect of a selection of ODD aspects, obtaining at least one concept comprising essential features of the ODD aspect via a user interface, creating a collection of images depicting the essential features of the concept, determining a score for each concept with respect to a target class (kt) learned by the Al model by applying a concept-based explanation method to the Al model for each class of the Al model, which is relevant for the concept and for each concept, and outputting the ODD coverage metric represented by the determined scores for each ODD aspect, via a user interface.

[0015] This approach allows to characterize the ODD aspect by one or more concepts and to provide a quantified measure of the performance of the Al model with respect to these concepts, i.e., ODD aspects. The concept comprises essential features of the ODD. The concept-based explanation method utilizes these high-level human-interpretable concepts to ascertain if they are acquired by the Al model on a global level. This is advantageous since this approach enables humans and the model to communicate in the same language, which renders generated explanations easy to interpret. The calculation of the scores can be automated and the whole process is semi-automated.

[0016] A second aspect concerns an apparatus comprising at least one processor and configured to perform the above-mentioned method.

[0017] A third aspect concerns a computer program product comprising instructions which, when the program is executed by one or more computers, cause the one or more computers to carry out the steps of the method according to performs the method as described above.

[0018] The computer program product can be stored on a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps as described before. In an advantageous embodiment the Al model is a classification model trained to classify images into classes, or a semantic segmentation model which is modified into a classification model. Thus, the method can be applied to a wide variety of Al models.

[0019] In an advantageous embodiment images of the concepts are generated by a generative model based on the objects and attributes of the ODD aspects which are relevant for the target class. Since images are required to create concepts, obtaining them is the main limiting factor of concept-based explanation methods. Advantageously, generative models are able to automatically generate a large number of images depicting the predefined feature in different variations based on inputting the predefined features.

[0020] In an advantageous embodiment a high value of the score indicates that the Al model identifies the features of the ODD aspect with high probability.

[0021] Thus, the score of a concept indicates the importance of the concept for the classification task and provides a human-interpretable measure for the quality of the performance of the Al model.

[0022] In an advantageous embodiment, if the ODD aspect correlates to more than one class of the Al model, the score for the concept created for this ODD aspect is determined by averaging the scores determined for all concept-target class combinations being relevant for the ODD aspect. If the ODD aspect is more specific with respect to objects and attributes than a single target class, the score indicates that the Al model (Me) learns the concept being included in the target class (kt). Further on, if the ODD aspect and the respective concept scores for all relevant classes is low and / or is not significant, calculating an intersection-over-union score can be used as a fallback score calculation.

[0023] Thus, a score can be determined for each ODD aspect independent of a precise matching of the scope of the ODD aspect and the target class.

[0024] In an advantageous embodiment the concept-based explanation method is only performed upto the layer of the classification model which is most relevant for classifying the image into the considered target class. This reduces the processing capacity required to determine the scores.

[0025] In an advantageous embodiment the concept-based explanation method is called “Testing with Concept Activation Vectors” (TCAV) algorithm. Only low programming effort is required to adapt the known TCAV method for the present object. TCAV provides valuable scores and therefore metric for the Al model covering the considered concept. In an advantageous embodiment a pass-threshold value and / or a fail-threshold value is input for each of the ODD aspects and an indication to each ODD aspect whether the ODD coverage metric satisfies one of the thresholds is output via the user interface.

[0026] This enables to output an easily comprehensible overview of the ODD coverage metric of the considered ODD aspects.

[0027] In an advantageous embodiment of the apparatus the user interface is configured to select, add, modify, weight concepts to emulate the ODD aspect. Thus, an adaption of the concepts is easily possible. The user interface allows to refine the concepts which emulate the various ODD aspects. Thus, the coverage of the ODD aspects by the considered Al model can be refined and deficiencies of the Al model can be revealed and measures to improve the performance of the Al model can be derived.

[0028] The invention will be explained in more detail by reference to accompanying figures. Similar objects will be marked by the same reference signs. The figures are provided to illustrate example embodiments described herein and are not in-tended to limit the scope of the disclosure.

[0029] Brief Description of the Drawing

[0030] FIG. 1 depicts an embodiment of an Operational Design Domain (ODD) taxonomy with several levels.

[0031] FIG. 2 illustrates an embodiment of the steps of the inventive computer-implemented method in form of a flow chart.

[0032] FIG. 3 schematically illustrates steps for extending a segmentation model into a classification model.

[0033] FIG. 4 illustrates the contribution of each layer of the Al model to the different concepts of an ODD aspect for determining a score by the concept-based explanation method.

[0034] FIG. 5 illustrates steps for determining an Intersection-over-Union (loU) score.

[0035] FIG. 6 illustrates a process for calculating Intersection-over-Union score in form of a flow chart. FIG. 7 schematically illustrates an embodiment of the inventive apparatus.

[0036] FIG. 8 illustrates an embodiment of the output displayed on a user interface of the apparatus.

[0037] Description of Examples

[0038] It is noted that in the following detailed description of embodiments, the accompanying drawings are only schematic, and the illustrated elements are not necessarily shown to scale. Rather, the drawings are intended to illustrate functions and the co-operation of functions or components. Here, it is to be understood that any connection or coupling of functional blocks, devices, components or other physical or functional elements could also be implemented by an indirect connection or coupling, e.g., via one or more intermediate elements. A connection or a coupling of elements or components or nodes can for example be implemented by a wire-based, a wireless connection and / or a combination of a wire-based and a wire-less connection.

[0039] Functional units or modules can be implemented by dedicated hardware, e.g., processor, firmware or by software, and / or by a combination of dedicated hardware and firmware and software. It is further noted that each module and functional unit described for an apparatus can perform a functional step of the related method and vice versa.

[0040] The numeration of the steps in the methods are meant to ease their description. They do not necessarily imply a certain ordering of the steps. In particular, several steps may be performed concurrently.

[0041] In the following, embodiments of the invention will be described in detail with reference to the accompanying drawings. It is to be understood that the following description of embodiments is not to be taken in a limiting sense. The scope of the invention is not intended to be limited by the embodiments described hereinafter or by the drawings, which are taken to be illustrative only.

[0042] To ensure more trust and safety in Artificial Intelligence (Al) models applied in a safety critical system like an autonomous vehicle (AV), limiting constraints are defined to describe under which conditions the Al model is expected to function as intended. These constraints are described by an Operational Design Domain (ODD) which specifies ODD aspects, i.e. , environmental objects and attributes with static and dynamic parameters the safety critical system must be able to manage. Examples of such objects are single rail tracks without curves, double tracks, no train driving towards position, persons, cars, signs poles, bridge, trees.

[0043] Example of such attributes are no visibility limitation due to weather or time of day, expect all possible traffic participants, expect animals.

[0044] Fig. 1 shows an exemplary embodiment of an ODD 30 provided by a taxonomy for ODD documents structured in 3 levels of granularity. ODD 30 represents the ODD in general, i.e., level 0. The ODD 30 divides on level 1 into three ODD aspects, which are “Scenery”, “Environmental Conditions” and “Dynamic Elements”. For instance, the operational design domain aspect of level 1 “Scenery” can comprise basic structures such as tracks, bridges, or the intersection of a rail. The ODD aspect “Environmental conditions”, see reference sign 31 may comprise on level 2 attribute like “illumination”, see reference sign 32, “connectivity” and “weather”. The level 2 attribute are further detailed on level 3. E.g., level 2 attribute “Illumination” comprises several level 3 attributes like lighting, cloud or moon elevation, see reference sign 33. Finally, another ODD aspect is called “Dynamic elements” which comprises objects like pedestrians, cars, trains, or any other vehicle or agent.

[0045] An embodiment of the steps of the inventive computer-implemented method are depicted in form of a flow chart in Fig. 2. The method automatically determines an Operational Design Domain (ODD) coverage metric for an Al model applied in a safety critical system. The following steps are performed for each ODD aspect of a subset of ODD aspects, wherein a subset include one up to all ODD aspects of the ODD. The ODD aspect can be an ODD aspect of each ODD level, i.e., of different granularity with respect to object and attribute.

[0046] In a first step 10 at least one concept C1 , .., On is obtained comprising essential features of the ODD aspect via a user interface via a user interface. One example for a concept to identify rails are parallel lines. In step 11, a collection of images Im (C1 ,.., On) is created depicting the essential features of the concept. In our example concept of “parallel lines”, a multitude of images are created which show parallel lines extending in different direction, which are bend, and further. In step 12 a score is determined for each concept with respect to a target class learned by the Al model. The score S (C1,.., On) is determined by applying a concept-based explanation method to the Al model for each class of the Al model, which is relevant for the concept. The score of the concept is the importance of the concept for the classification task of the Al model. This is performed for each target class learned by the Al model, i.e., for each combination of concept and class. The Al model is trained to perform a classification task, i.e., to classify an input image into one of several classes. At least the ODD coverage metric represented by the determined scores S (C1 Cn) are output for each ODD aspect, via the user interface, see step 13.

[0047] As an example, for the ODD aspect “Scenery” which includes “rail tracks”, “bridges” and “intersection of a rail” a concept for rail tracks is generated comprising images depicting rail tracks in various environments, a concept for bridges is generated containing images of different kinds of bridges, and a concept for intersection of rail containing images of intersection of rail is generated depicting different types of rail intersections.

[0048] The method is proposed to determine an ODD coverage metric for classification model Me. As an option, the concept-based explanation method can also be used by an image segmentation model Ms to test if and how good the ODD has been learned. The following steps describe the process to extend the segmentation model Ms to classification model Me, see Fig. 3.

[0049] Contrary to classification models Me, a semantic segmentation model Ms outputs a tensor 22 with shape [H, W, C] where H and W are the height and width values of the input image 21 respectively and C is the number of output classes. This means instead of getting one scalar per class, the segmentation model Ms gives one [H, W, 1] tensor per class. This tensor is made of the raw, i.e., nonnormalized network output values that the segmentation model Ms generates for each pixel for one class, also referred as logits. To turn these tensors into scalar values, these logit values can be aggregated (Ms is not changed) resulting in classification model Me. This aggregation is done by only averaging the values where the pixel is labeled as the corresponding class. To determine a label, the maximum value among classes is used. That way, the obtained scalar value, in the end, will be composed of the average logit values of pixels that belong to the corresponding class. Otherwise, the value will simply be omitted while calculating the average. After all, it doesn’t make sense to get the contribution of class k1 for a pixel that is part of class kn.

[0050] Fig. 3 shows an aggregation of target pixel values per channel. Target pixels are those with the greatest activation values among all channels for a class. In Fig. 3 target pixels for the class “dog”, see 23, are depicted in dark color which are then averaged and make up the final scalar 24. The same goes for other classes like “background” 25, 26.

[0051] The steps of an exemplary embodiment of the computer-implemented method are described below in more detail. First, a target class kt to be explained of the Al model Me is selected out of the classes which are relevant for the concept. Images of objects of the target class kt are gathered. I.e., these target images are images with are / should be classified into the target class kt by the Al model. These images will be referred as target images.

[0052] Next, concepts are derived from the ODD aspect which might be relevant for target class kt. Each concept comprises a multitude of images which depict the concept. Generative models, also called Generative Al models, are used to generate the images for the concepts. Even though these generative models create satisfactory images, the generative models would in most cases not generate the desired output images for concepts including additional ODD “Level 3” constraints. Even if the generated image looks feasible to a human, there is no way to measure specific scenarios like strength of the wind or temperature by looking at a single image. However, for more general definitions, e.g., windy day without a specific wind strength or a pedestrian without a specific height, generative models can be used to create realistic-looking concept images and report back a metric for an ODD aspect.

[0053] In the next step, a score is determined for each concept with respect to the target class kt by applying a concept-based explanation method to the Al model, which is relevant for the concept. One concept-based explanation method is “Testing with Concept Activation Vectors”, in short TCAV, described by Kim et al, see https: / / doi.org / 10.48550 / arXiv.1711.11279. In the described embodiment when TCAV is applied as concept-based explanation method, but also other embodiments of concept-based explanation methods are applicable. TCAV determines whether the Al model identifies certain concepts, e.g., rails are detected because parallel lines have been detected. TCAV is so far only applied to classification models and simple structured concepts. TCAV only provides information for one concept, but not for several segments in combination with each other. In this embodiment TCAV is applied, but

[0054] The concept-based explanation method, e.g., TCAV, is performed for the target class kt with all concepts C1, ... , Cn of the Al model Me. The concept-based explanation method is performed either for all layers of the Al model or for all layers until a bottleneck layer is determined or for bottleneck layers only. A bottleneck layer is a layer in a neural network that contains few nodes compared to the previous layers. It represents the input with reduced dimensionality. After conceptual sensitivity is calculated for an image that belongs to the class to be explained, this procedure is done for all other provided target class images. In the end, the TCAV score of a concept is the fraction of pictures that were sensitive towards that concept provided by

[0055] The score resulting of the concept-based explanation method, e.g., TCAV, provides for class kt the importance of all concepts C1, , Cn for all calculated layers of model Me.

[0056] An example of determined TCAV scores is depicted in Fig. 4. A semantic segmentation model Ms utilizing a UNet architecture is trained to detect obstacles and rails in each image. An obstacle can be anything standing on the tracks. Here it is checked if the model Ms really pays attention to the two most important objects on the tracks since ‘Vehicle (Car)’ and ‘Human (Pedestrian)’ are likely to be in the ODD.

[0057] Diagram 43 shows TCAV scores S for class target class “obstacles”. Images 40 are gathered for the target class “obstacles”. A concept is defined for “vehicle_on_rail” and respective images 41 are generated. Another concept is defined for “human_on_rail” and respective images 42 are generated. Diagram 43 shows for concept “vehicle_on_rail” 44 high scores only for layer L1 and very low scores for layers L2 to L4 of the Al model. The scores for concept “human_on_rail” 45 show high scores for layer L1 and strong decreasing scores for layers L2 to L4 of the Al model. The TCAV scores S indicate that the Al model’s bottleneck layer (L4), does not really associate an obstacle with a vehicle standing in its way. The same is also true for humans. This is a strong indicator that the Al model is not covering the ODD, hence it is not ready for deployment.

[0058] When applying concept-based explanation method, e.g., TCAV, for the described purpose, the definition of the explained target class with respect to the concept plays a tremendous role. Ideally, TCAV is used to explain a more abstractly defined class using more granularly defined concepts. However, this might not always be the case for the Al model’s pre-defined target classes and the system’s ODD aspects.

[0059] Three cases of hierarchical levels between of target class(es) kt and ODD aspect are identified: a. Level (Target Class) > Level (ODD aspect), i.e. , the ODD aspect / concept is more specific than the target class kt:

[0060] In this case, the target class being explained is defined more generally than the tested ODD aspect. For example, the target class is “Vehicle’, and the ODD aspect is ’Car’. Simply applying the TCAV algorithm to the Al model Me will reveal that if the Al model Me learns cars to be vehicles. b. Level (Target Class) = Level (ODD aspect), i.e., the ODD aspect is the same as target class:

[0061] This scenario is roughly the same as the first case, except now the target class shares the same definition as the ODD aspect. One example is explaining ‘Building’ class of the Al model Me with the ODD aspect “Buildings”. This means that the Al model is expected to work as intended when there are buildings in the scene. In an embodiment of the method, to test the robustness of the Al model for different buildings, a concept ‘Building’ is created via a user interface, with images of multiple different types of buildings and the TCAV score is calculated. When the model outputs a high score, this indicates that the Al model learned the essentials of what makes an object, a building. A high value of the score indicates that the Al model identifies the features of the ODD aspect with high probability. c. Level (Target Class) < Level (ODD aspect), i.e., the ODD aspect is less specific than the target class:

[0062] It is also possible that an ODD aspect correlates to more than one target class kt. For example, one target class contains ‘child’ another target class contains ‘person’, but the ODD aspect indicates only ‘Pedestrian’. In such cases, to cover the one ODD aspect, an average TCAV score of all relevant target classes is output. Meaning we must test all classes which are considered as pedestrians. Otherwise, it would be only possible to report back a partial coverage score for an ODD aspect. I.e., if the ODD aspect correlates to more than one class of the Al model, the score for the concept created for this ODD aspect is determined by averaging the scores determined for all concept-target class combinations being relevant for the ODD aspect.

[0063] Some ODD aspects may be relevant for all target classes such as ODD aspect ‘Environmental Conditions’ of an ODD. In these cases, the concept capturing the ODD aspect will not be closely correlated with any of the target classes. I.e., if the ODD aspect and the respective concept is relevant for all classes of the Al model, the score for the concept with respect to each class is low and / or is confirmed by calculating an intersection-over-union score. Here, it is still beneficial to use the concept-based explanation method to see that there’s indeed no correlation between the target class and the concept. However, this does not inform the user about whether the ODD aspect is covered or not. An approach to prove that the ODD aspect is covered is depicted in Fig. 5. First images of the concept are generated and labelled with ground truth, see 50. The prediction for a class of the Al model Me is determined for each image, see 51. Then, an “Intersection over Union”, short loU, also named Jaccard index, is calculated for all images for a target class, see 52. I.e., the intersection of the location of the object determined by the Al model Me, see dashed frame, and the actual location of the object in the labelled image, see solid frame, is determined. Subsequently a “class loU” (cloU) is calculated for the class as average of all concept images, see 53. Finally, a “mean loU” is determined by averaging every “class loU” for the whole concept, 54. The intersection over union is a statistic used gauging the similarity between finite sample sets like images.

[0064] The output ODD coverage metric for ODD aspects is at least one of scores determined by the concept-based explanation method, scores determined by the intersection-over-union scores, or an average of intersection over union scores and the concept-based explanation method scores. In the following some examples are described how the coverage of ODD aspects by the Al model is evaluated and by which approach the score, i.e., the ODD coverage metric is determined.

[0065] A concrete example of ODD aspects and the evaluation of the ODD coverage are described in the following. It is assumed that the Al model Me is trained to classify input images into classes “car”, “pedestrians”, “rail”, “building” and “vegetation”.

[0066] ODD aspect: Environmental Condition -> Weather -> Sunny

[0067] Having this ODD aspect means that a safety critical system is expected to work as intended under sunny weather. That means all the classes of Me should have high accuracy. To measure this, a simple mean loU (intersection over union) metric over all classes is sufficient.

[0068] ODD aspect: Scenery -> Zone -> Vegetation

[0069] The ODD aspect is the same as one class of the Al Model Me. A concept called ‘Vegetation’ is generated which contains different types of foliage or trees. Now, if the vegetation class obtains a high TCAV score for Vegetation concept’ this ensures that the Al model learned the right concept behind the vegetation object.

[0070] On top of this, if images in ‘building concept’ can be annotated with ground truth, we can have loU metric for building class (not the mean loU over all classes) additionally. When the ODD aspect is the same or more general than segmentation model’s target classes, we can make use of TCAV and loll for that class.

[0071] ODD aspect: Scenery -> Drivable Area -> Highway

[0072] This ODD aspect is not captured by any class of Al model. It is impossible to cover this requirement with our current model architecture. Both loU and TCAV don’t help.

[0073] In this case no score can be provided via a user interface to a user, e.g., an Al model designer. This output indicates that the ODD aspect is not covered by the Al model and further steps to amend the Al model can be initiated.

[0074] ODD aspect: Dynamic Elements -> T raffic Agent -> Pedestrian -> Adult and Child

[0075] This ODD aspect is of level 3, but concept images can still be created with the generative model. The Al model Me has a class “pedestrian” that is more general than ODD aspect. The score is determined by applying the concept-based explanation method and output as ODD coverage metric for this ODD aspect.

[0076] ODD aspect: Dynamic Elements -> Subject Vehicle

[0077] Here, the target class ‘car’ is more generally defined as the ODD aspect just stands for ‘vehicle’. As there is no class for each sub-definition of ‘vehicle’ class, only the class ‘car’ is tested, and a partial coverage score for this ODD aspect is output.

[0078] Fig. 6 illustrates the conditions related to calculating an Intersection-over-Union score. Whether an loU score is the appropriate metric to be determined for an ODD aspect has to be evaluated for each ODD aspect individually. This process can be applied for any ODD aspect of each ODD level.

[0079] At first, the ODD aspect under evaluation is selected, see 60. Then it is checked whether a generative model can generate images for the ODD aspect, see 61 . If yes, use the generative model to create images, see 62. If no images can be created by the generative model, collect images from the real world, see 63.

[0080] Next, it is checked whether the ODD aspect affects all target classes of the inspected Al model Me, see 64. If this is the case, the generated images are labelled for all target classes of the Al model Me and calculate the mean loU, see 65 and the process is finished. In case the ODD aspect does not affect all target classes, it is determined whether the level of the target class is higher than the level of the ODD aspect, see 66. If yes, the ODD aspect can only be partially checked, see 67. If no, the ODD aspect can be checked using a score by applying concept-based explanation method, see 68. In both cases, see 67 and 68, it is checked, whether the images have ground-truth labels, see 69. If no ground-truth labels are available, no loll score is determined and the process ends, see 71. In case ground-truth labels are available for the images loU score is calculated for the target class in addition to the score by applying concept-based explanation method, e.g., TCAV score, see 70 and the process ends, see 71.

[0081] At least an ODD coverage metric is output either representing for a single ODD aspect and / or for an average metric for the entire ODD coverage. A single ODD-coverage score is output under following conditions.

[0082] In scenarios where concept-based explanation method is applied, conceptual (sensitivity) scores can be obtained. Additionally, whenever ground-truth labeling is present for the images, either concept images or target images for the target class, intersection over union scores are calculated. In the end, the final metric to be output can be either of these scores or a weighted or unweighted average of both when applicable.

[0083] Sometimes focusing on one single target class can be sufficient as score for the ODD aspect. For instance, when the ODD aspect is an environmental condition, the scores of all target classes should be considered because weather conditions will affect all of them. In that case the reported ODD coverage metric would be the aggregation of all Al model classes. On the other hand, to test if the Al model is working well while detecting children, the scores generated for ‘Pedestrian’ class would be sufficient. In this example, reporting back a combination of loU and TCAV score of ‘Pedestrian’ class would be the determining factor for the ‘Children’ ODD aspect.

[0084] After all the ODD aspects are evaluated a global coverage score can be generated. To do so, the single scores are simply aggregated, or a PASS threshold value and / or FAIL threshold value is set for all ODD aspects. In the end, it is output how many of the ODD aspects of the different levels are satisfied, e.g., by providing a percentage value for the deployment readiness status of the Al model and the safety critical system respectively.

[0085] Fig. 7 shows an apparatus which is configured to perform the above-described method. The apparatus comprises a storage medium and at least one processor. The apparatus 70 comprises a user interface 71 , an image unit 72 and a score processing unit 73. The user interface 71 is configured to obtain at least one concept comprising essential features of the ODD aspect. The user interface 71 is configured to output the ODD coverage metric represented by the determined scores for each ODD aspect. The image unit 72 to create a collection of images depicting the essential features of the concept. The score processing unit 73 is configured determine a score for each concept with respect to a target class learned by the Al model by applying a concept-based explanation method to the Al model for each class of the Al model, which is relevant for the concept, and for each concept.

[0086] An embodiment of the output displayed on the user interface 71 is depicted in Fig.8. In an ODD zone 81 the considered ODD aspects are displayed, preferably structured according to the ODD level. A pass / fail indication of the respective ODD aspect is displayed at the right side of ODD zone 81 indicating whether a predefined pass-threshold value and a fail-threshold value has been met. In the concept zone 82 the considered concepts are displayed accompanied by a pass / fail indication. The ODD coverage metric is output and displayed in relevance zone 83, indicating its current value 85 and a target value 84. The pass-threshold value and / or a failthreshold value for each of the ODD aspects can be input, e.g., via an input zone 85 and outputting an indication to each ODD aspect whether the ODD coverage metric satisfies one of the thresholds in ODD zone 81. Further the user interface 71 is configured to select, add, modify, weight concepts to emulate the ODD aspect. The respective input mode can be selected in input zone 85. Thus, an interaction with the user is possible during the ODD coverage metric determination is possible enabling a human-in-the-loop approach.

[0087] The proposed ODD coverage metric will give a quantifiable result on deployment readiness of an Al model applied in a safety critical system. The calculation is automatically performed and the whole process is semi-automated. With pure manual testing, it is both more tedious and less reliable to have an understanding about ODD coverage. An optional human assessment of the metric and feedback of resulting measures is possible with the disclosed approach.

[0088] Main benefits are an automatic ODD metric generation, semi-automatic ODD coverage analysis, more efficient, time-saving ODD coverage analysis, quantitative metrics for certification process. Human understandable explanations make it easier for the certification party or the developer to assess the safety case.

[0089] All features described and / or drawn can be advantageously combined with each other within the scope of the invention. The invention is not limited to the embodiments described.

Claims

Claims1. A computer-implemented method for automatically determining an Operational Design Domain (ODD) coverage metric for an Al model applied in a safety critical system of an autonomous vehicle wherein the Operational Design Domain specifies different ODD aspects concerning environmental objects and attributes the Al model must be able to identify, comprising: for each ODD aspect of a selection of ODD aspects, obtaining (10) at least one concept (C1 , ... , On) comprising essential features of the ODD aspect via a user interface, creating (11) a collection of images depicting the essential features of the concept, determining (12) a score for each concept with respect to a target class (kt) learned by the Al model (Me) by applying a concept-based explanation method to the Al model (Me) for each class of the Al model (Me), which is relevant for the concept (01 , ... , On), and for each concept, and outputting (13) the ODD coverage metric represented by the determined scores for each ODD aspect, via the user interface.

2. A computer-implemented method according to claim 1 , wherein the Al model is a classification model (Me) trained to classify images into classes (k1 , ..., kn), or a semantic segmentation model (Ms) which is modified into a classification model (Me).

3. A computer-implemented method according to any of the preceding claims, wherein images of the concepts are generated by a generative model based on the objects and attributes of the ODD aspects which are relevant for the target class (kt).

4. A computer-implemented method according to any of the preceding claims, wherein a high value of the score indicates that the Al model identifies the features of the ODD aspect with high probability.

5. A computer-implemented method according to any of the preceding claims, wherein if the ODD aspect is more specific with respect to objects and attributes than a single target class, the score indicates that the Al model (Me) learns the concept being included in the target class (kt).

6. A computer-implemented method according to any of the preceding claims, wherein if the ODD aspect correlates to more than one class of the Al model (Me), the score for the concept (C1, ... , Cn) created for this ODD aspect is determined by averaging the scores determined for all concept-target class combinations being relevant for the ODD aspect.

7. A computer-implemented method according to any of the preceding claims, wherein if the ODD aspect is relevant for all classes of the Al model (Me), the score for the concept (C1 , ... , Cn) with respect to each class is low and / or is confirmed by calculating an intersection- over-union score.

8. A computer-implemented method according to claim 7, wherein calculating the intersection- over-union score comprises labelling the images of the concept (C1, ... , Cn) and determining Intersection-over-union (doll) measures of each of the target classes (k1, ..., kn) individually for all images and obtaining a mean loll by averaging each target class loU value resulting in one final score for the coverage of the ODD aspect.

9. The computer-implemented method according to any of claims 7 or 8, wherein the output ODD coverage metric is at least one of- the scores determined by the concept-based explanation method,- the scores determined by intersection-over-union,- an average of intersection-over-union-scores and the concept-based explanation method scores.

10. A computer-implemented method according to any of claims 2-8, wherein the conceptbased explanation method is only performed up to the layer of the classification model (Me) which is relevant for classifying the image into target class kt.

11. The computer-implemented method according to any of the preceding claims, wherein the concept-based explanation method is a Concept Activation Vectors (TCAV) algorithm.

12. The computer-implemented method according to any of the preceding claims, wherein via the user interface, inputting a pass-threshold value and / or a fail- threshold value for each of the ODD aspects and outputting an indication to each ODD aspect whether the ODD coverage metric satisfies one of the thresholds.

13. Apparatus (70) comprising at least one processor configured to perform an automatic determination of an Operational Design Domain (ODD) coverage metric for an Al model applied in a safety critical system of an autonomous vehicle, wherein the Operational Design Domain specifies different ODD aspects concerning environmental objects and attributes the Al model must be able to identify, configured to perform the steps: for each ODD aspect of a selection of ODD aspects, obtaining (10) at least one concept (C1 , ... , On) comprising essential features of the ODD aspect via a user interface, creating (11) a collection of images depicting the essential features of the concept, determining (12) a score for each concept with respect to a target class (kt) learned by the Al model (Me) by applying a concept-based explanation method to the Al model (Me) for each class of the Al model (Me), which is relevant for the concept (C1 , ... , On), and for each concept, and outputting (13) the ODD coverage metric represented by the determined scores for each ODD aspect, via the user interface.

14. Apparatus according to claim 13, wherein the user interface (71) is configured to select, add, modify, weight concepts to emulate the ODD aspect.

15. A computer program product comprising instructions which, when the program is executed by one or more computers, cause the one or more computers to carry out the steps of the method according to any one of claims 1 to 12.

Images