Vehicle, device, computer program, method for checking a machine learning-based model for an error, machine learning-based model and use thereof

Abstract

Embodiments of the present disclosure relate to a vehicle, a device, a computer program, a method for checking a machine learning-based model for an error, a machine learning-based model, and the use thereof. The model comprises obtaining, for a plurality of samples, a concept activation vector (CAV) of the model for segmenting a respective sample and obtaining a respective ground truth segmentation for the samples. Further, the method comprises adapting the model based on a comparison of the CAVs and the ground truth such that a deviation of a projection of the samples using the CAVs and the ground truth is reduced.The method also provides for clustering the CAVs into one or more clusters based on a semantics of concepts of the CAVs, obtaining grouping rules for classifying objects based on the clusters, and checking the model for errors based on the grouping rules.

Description

[0001] Embodiments of the present disclosure relate to a vehicle, a device, a computer program, a method for checking a machine learning-based model for an error, a machine learning-based model, and the use thereof. In particular, but not exclusively, the present disclosure relates to a concept for checking models for object classification, e.g., in automotive applications. Fault detection in ML models

[0002] Some tracking programs for object detection use an original network to evaluate the accuracy of object tracking and train an auxiliary network to predict tracking accuracy using the activation functions from the intermediate layer. Furthermore, context (temporal information, activations, and raw pixels / optical flow) is fed into the perturbation detection network. The perturbation detection network predicts the accuracy of the perturbation of object tracking from the activations and context.

[0003] Another architecture for object tracking provides anomaly prediction using two modules. The first module (correlation filter) takes an original image and the object's hull as input and outputs a correlation map reflecting the similarity of the target based on the object and context. The input to the second module, called anomaly tracking, is the correlation map from the previous module, and the output is a probability prediction of the tracking accuracy, ranging from 0 to 1.

[0004] Another approach uses a concept-based object recognition engine to recognize complex objects using simpler ones, which are structural parts of the complex. The goal is to analyze user queries (speech, text, etc.) and select the appropriate object recognition engine for the given task: excellent object recognition, concept-based recognition models with known classes, and with unknown classes. The framework is not designed to identify failure cases.

[0005] Another framework is designed to process queries on known and unknown classes in the case of a known object class and uses a known detection model to detect the query object. Alternatively, a neural network can segment potential objects and generate an object vector for each object, generate a query object vector, generate a correlation score between the vectors, and assign the object based on the correlation score. The query can be a text string or a selection in an image.

[0006] Another approach provides a method for overview monitoring for detecting an anomaly occurring in the area, preventing false detection caused by changes in an environmental condition, and including means for photographing a monitoring area on a road with one or more viewing angles and object extraction means for extracting a range of an object occurring in the monitoring area and a pixel value from a video captured by the photographing means.The interference detection system includes an object detection means for identifying a type of the object from a set of local features by dividing the area of ​​the object and the pixel value detected by the object extraction means into blocks based on the view angle and a set of determination criteria for positions in the video, and an interference detection means for detecting the presence or absence of an obstacle in the video from information about the type of the object detected by the object detection means.

[0007] Another approach proposes a mechanism for determining a success rate, which indicates the success of segmenting a 3D image. The mechanism proposes obtaining one or more 2D images of different target views of a target object in the 3D image by processing a segmentation result of the 3D image. (A view of) each 2D image is classified using an automated classifier. The classification results are used to determine a success rate, which indicates, for example, whether or how closely the 3D segmentation result represents a segmentation result based on verified data (ground truth) with sufficient accuracy, e.g., for medical decision-making.

[0008] Another approach proposes to transform an opaque neural network into an explainable neural network, for example by extracting rules to explain the network's decisions.

[0009] Some of the approaches use supervised methods. However, they may require labor-intensive pixel-by-pixel segmentation or pixel sampling.

[0010] In contrast, unsupervised (human) methods can retrieve uninterpretable concepts.

[0011] In concept bottlenecks, only a single layer remains interpretable and bottleneck neurons are placeholders that may not correspond to the real meaning.

[0012] Some approaches may make it impossible to directly visualize representations of concepts.

[0013] Other approaches propose to generate a single-vector representation of the concept, which may be distributed in different regions of the feature space (i.e., the concept may be fragmented in the feature space due to various factors).

[0014] Thus, there may be a need for an improved approach to verifying a machine learning-based model.

[0015] This need can be met by the subject matter of the independent claims.

[0016] In particular, the present disclosure provides a cost-effective strategy for generating sample concept representations and validating a machine learning-based model.

[0017] The proposed approach also enables internal representations in multiple layers, making the network transparent beyond a single bottleneck. Furthermore, it enables the generation of anomaly explanation map, making the review understandable for a human user.

[0018] The proposed approach also allows for the discovery of multiple or ideally all possible domains related to a concept by a sample analysis of inputs instead of a generalized optimization of sample sets.

[0019] Embodiments of the present disclosure provide a method for testing a machine learning-based model for error. The model comprises obtaining, for a plurality of samples, a concept activation vector (CAV) of the model for segmenting a respective sample and obtaining a respective ground truth segmentation for the samples. In the context of the present disclosure, the CAV is also understood as a numerical representation of the concept (CNR) or a vector representation of the concept (CVR). Furthermore, the method comprises adapting the model based on a comparison of the CAVs and the ground truth such that a deviation of a projection of the samples using the CAVs and the ground truth is reduced.The method also provides for clustering the CAVs into one or more clusters based on a semantics of concepts of the CAVs, obtaining grouping rules for classifying objects based on the clusters, and checking the model for errors based on the grouping rules.

[0020] In this way, the proposed approach provides a transparent solution for verifying a machine learning-based model.

[0021] Furthermore, the proposed approach is applicable to multiple layers of the model. Accordingly, some embodiments of the method include obtaining the CAVs and grouping rules for multiple layers of the model, where the grouping rules are layer-specific, as well as obtaining generalized grouping rules for multiple layers based on the layer-specific grouping rules. This enables more generalized detection of errors or faults for multiple layers.

[0022] In some embodiments, the method further comprises reducing a dimensionality of the CAVs for improved visualization quality of defects.

[0023] In some applications, the method may further include validating the model based on a comparison of the grouping rules and other CAVs generated when applying the model to a test sample. This allows for the identification of any errors or faults in the validated model, and the validated model can be adjusted as appropriate.

[0024] Accordingly, the method may further comprise adapting the model based on the identified error.

[0025] Further embodiments of the present disclosure provide a machine learning-based model obtainable using the method proposed in this document.

[0026] Embodiments may also provide for the use of such a machine learning-based model for a driver assistance system and / or a (semi-)autonomous driving system.

[0027] Further embodiments provide a device comprising one or more interfaces for communication and a data processing circuit configured to carry out the proposed method and / or to apply the machine learning-based model proposed in this document.

[0028] Other embodiments provide a vehicle including the proposed device.

[0029] Further embodiments provide a computer program comprising instructions that, when the computer program is executed by a computer, cause the computer to perform the proposed method and / or apply the proposed machine learning-based model.

[0030] Furthermore, embodiments will now be described with reference to the accompanying drawings. It is noted that the embodiments illustrated by the aforementioned drawings merely show optional embodiments as an example, and that the scope of the present disclosure is by no means limited to the presented embodiments. Short description of the drawings Fig. 1 shows a flowchart schematically illustrating an embodiment of a method for checking a machine learning-based model for an error; Fig. Figure 2 schematically illustrates how CAVs are obtained according to the proposed approach; Fig. Figure 3 illustrates schematically how the model can be fitted for multiple samples; Fig. Figure 4 illustrates an example of a concept-based classification; Fig. Figure 5 schematically illustrates different dimensionalities of CAVs; Fig. Figure 6 schematically illustrates how rules for error checking of a machine learning-based model can be extracted; Fig. Figure 7 shows a flowchart that schematically illustrates how global declarations are determined; and Fig. 8 shows a flowchart schematically illustrating an embodiment of a device according to the present disclosure.

[0031] As previously stated, concepts for validating machine learning-based models may have various drawbacks. Therefore, the present disclosure proposes a solution to address and / or eliminate the identified drawbacks.

[0032] Further aspects and features of the proposed approach will now be discussed with reference to Fig. 1 described in more detail.

[0033] Fig. 1 shows a flowchart schematically illustrating an embodiment of a method 100 for checking a machine learning-based model for an error.

[0034] The method 100 includes obtaining 110, for a plurality of samples, a concept activation vector (CAV) of the model for segmenting a respective sample. The samples correspond to or include images, e.g., camera images.

[0035] In explainable AI, a concept refers to a higher-level abstraction or representation of the feature or pattern learned by the model during training. In other words, a concept (also called a "semantic concept") is a scalar or vector in the feature space of the deep neural network (DNN) that corresponds to a region of the input space (e.g., a portion of the image). A semantic concept corresponds to a natural language concept that can be described by a word or phrase (e.g., "yellow," "pedestrian head," "person," etc.). Other examples include "car," "truck," "bus," "bicycle," and / or "motorcycle." A concept scalar represents the concept assignment (strength, importance) of an individual concept for the current prediction.

[0036] For the purposes of the present disclosure, the CAV is understood as a vector in the direction of the activation values ​​of the set of examples of the respective concept. In other words, concept activation vectors (also referred to as "concept vectors") represent the direction or "center of mass" (centroid) of concepts in the feature space, depending on the implementation. The dimensionality of the concept vector also depends on the implementation.

[0037] Furthermore, the method 100 includes obtaining 120 a respective ground truth segmentation for the samples. The ground truth is understood as an example of an ideal segmentation for the samples, e.g., an ideal segmentation of an image for the concept "bicycle" or "motorcycle." In practice, the ground truth segmentation can be determined manually, (semi-)automatically, and / or from a sample database that provides not only the samples but also corresponding examples of an ideal segmentation for one or more concepts.

[0038] The method 100 also includes adapting 130 the model based on a comparison of the CAVs and the ground truth, such that a deviation of a projection of the samples using the CAVs and the ground truth is reduced. For this purpose, for example, the loss function of the model, which indicates the deviation, is reduced or ideally minimized. Various machine learning techniques, e.g., supervised learning, can be applied using the samples and the ground truth as training data.

[0039] Furthermore, the method 100 proposes clustering 140 the CAVs into one or more clusters based on the semantics of the concepts of the CAVs. For example, CAVs that have similar or the same semantics are clustered. For example, CAVs related to two-wheelers, such as the CAVs for the concepts "bicycle" and "motorcycle," are clustered into the group "two-wheelers." Similarly, CAVs for vehicles that have four wheels may be grouped. Those skilled in the art having the benefit of the present disclosure will appreciate that various clustering techniques may be used for this purpose.

[0040] The method 100 also provides for obtaining 150 grouping rules for classifying objects based on the clusters. For this purpose, logic can be applied, for example. An example grouping is as follows: If the sample belongs to the group “two-wheelers” OR “four-wheelers”, then the concept is “vehicle”.

[0041] The grouping rules are defined, for example, by a user and / or using a natural language processing framework configured to understand and abstract the group labels.

[0042] Furthermore, the method 100 includes checking 160 the model for errors based on the grouping rules. This includes, for example, checking whether the grouping rules work for one or more test samples. For example, testing whether both motorcycles and cars are reliably classified as vehicles, e.g., that motorcycles and cars are identified as vehicles and are not confused with each other. The idea here is that the CAVs of cars and motorcycles must be similar, but still be grouped separately. If not, the model may be faulty.

[0043] In other words, the present disclosure proposes a method that helps human experts analyze and, if necessary, repair ML models, particularly, but not limited to, for the field of computer vision. The method can be used to detect errors in ML models using a rule base extracted from the ML model using so-called concepts. The method uses concept activation vectors to represent the concept in another layer of the ML model, which encodes the information about this concept in the latent space. The proposed method can be used to generate explanations (rules) for identifying potential errors in the model during its testing by a human expert. This allows the expert to test whether concepts that should be similar are also similar in the learned feature space of the ML model.The method also allows testing whether concepts are correctly distributed in the learned feature space (e.g., whether a class has multiple clusters). This makes it possible to assess whether an object (a class) has multiple appearances, such as pedestrians (e.g., different poses). Poorly learned feature spaces can also be identified, allowing a human expert to assess whether objects are similar or dissimilar. With the proposed method, a human expert can also verify whether existing model behavior rules are valid for the rules extracted from the ML model under test. Outliers can also be detected and their reasons for being present (e.g., two concepts are intermixed in a layered representation of the model).In summary, the proposed method enables a human expert to analyze an ML model for the mentioned errors using rules or equivalent representations.

[0044] In particular, the method can be applied to models of various applications, e.g., automotive or medical applications.

[0045] Further details and aspects (especially regarding concepts and explainable AI) are discussed below with reference to Fig. 2 described.

[0046] Fig. Figure 2 schematically illustrates how CAVs are obtained according to the proposed approach.

[0047] As mentioned previously, the concept activation vector (CAV) is a projection vector containing a set of weights for deep neural network (DNN) units (e.g., convolutional neurons or convolutional filters). When an optimal CAV is multiplied by a tensor of the activation function of the network's hidden layer, it provides the optimal activation pattern corresponding to a ground-truth segmentation mask (see Fig. 2).

[0048] Based on this insight, the idea of ​​the proposed approach is to find neural units that contain the information relevant to the desired segmentation mask. This information is (can be) encoded in different units in each layer. It is proposed to find these NN units by optimizing weights and calculating the difference between the activation projection and the desired segmentation (see Fig. 2) The process is as follows: An input image 220 is passed to the ML model 230 under test, resulting in some predictions 240. For selected layers 232 of the ML model 230, the activations are multiplied by a CAV 250, which is optimized using a loss function 260, e.g., Dice loss, mean absolute error (MAE), mean squared error (MSE), Jaccard coefficient (IoU - Intersection over Union), and / or the like. The loss describes the difference between the ground truth mask and the projected mask obtained from the multiplication of the CAV and the activations of a layer.

[0049] It is proposed to use the process of CAV optimization for multiple samples (see Fig. 3), which may belong to different classes, to obtain a set of CAVs (ground-truth CAVs) that can be used in future analysis to generalize the knowledge gained from many samples. These CAVs can be used as local explanations, for comparing the response of different networks to a single sample, etc.

[0050] For the same semantic concepts or objects, each DNN layer can have one or more units that are activated when a concept or object is presented in the input space (e.g., the object is present in the input image). However, due to the high semantic complexity of detectable objects (e.g., a "person" can be standing, sitting, walking, close to or far from the camera, etc.), the network (potentially) requires multiple groups of units (e.g., clusters) and activation patterns corresponding to an object.

[0051] To obtain labels (semantic groups) of optimized CAV groups, clustering can be used (see Fig. 6), which is generally used when searching for similar patterns in information. Clustering can be combined with prior dimensionality, internal reduction (may result in a loss of information but increases the visualization quality, see Fig. 5) or without them. After identifying the semantic groups, generalization can take place. Each of the inspected sample classes can be related to one or more similar semantic groups (corresponding to the activation of one or more NN units). These groups can be described in the form of logical rules or equivalent representations that explain all cases when a sample belongs to a class or not. Example: IF current_sample BELONGS TO semantic_group1 or semantic_group3 THEN class2. This can be inferred, for example, from the analysis of cluster label assignments (frequency of cluster labels and samples to which they were assigned). Alternative rule representations are also possible: probabilities, fuzzy logic rules, etc.

[0052] Rules generated for a single layer of a network describe only that layer. To further generalize the functioning of NNs, rules from multiple layers can be combined. This can help to recognize more complex patterns and generate global explanations of the model (see Fig. 7).

[0053] Disturbances in network operation can be detected by comparing the global explanation (multi-layer rules) with CAVs generated in inspected layers for a new sample.

[0054] Fig. Figure 8 shows a block diagram schematically illustrating an embodiment of such a device 800. The device comprises one or more interfaces 810 for communication and a data processing circuit 820 configured to carry out the proposed method.

[0055] In embodiments, the one or more interfaces 810 may comprise wired and / or wireless interfaces for transmitting and / or receiving communication signals in connection with the implementation of the proposed concept. In practice, the interfaces comprise, for example, pins, wires, antennas, and / or the like. The interfaces may also comprise means for (analog and / or digital) signal or data processing in connection with the communication, e.g., filters, samples, analog-to-digital converters, signal acquisition and / or reconstruction means, as well as signal amplifiers, compressors, and / or any encryption / decryption means.

[0056] The data processing circuit 820 may correspond to or include any type of programmable hardware. For example, examples of the data processing circuit 820 include a memory, a microcontroller, field-programmable gate arrays, one or more central and / or graphics processing units. To carry out the proposed method, the data processing circuit 820 may be configured to access or retrieve a suitable computer program for executing the proposed method from a memory of the data processing circuit 820 or a separate memory communicatively coupled to the data processing circuit 820.

[0057] In the foregoing description, it will be appreciated that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure should not be construed as reflecting an intent that the claimed examples require more features than are expressly recited in each claim. Rather, the subject matter may lie in fewer than all of the features of a single disclosed example, as the following claims reflect. Thus, the following claims are hereby incorporated into the specification, with each claim capable of standing on its own as a separate example.While each claim may stand on its own as a separate example, it is noted that although a dependent claim in the claims may refer to a specific combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent claim, or a combination of any feature with other dependent or independent claims. Such combinations are suggested herein unless it is stated that a specific combination is not intended. Furthermore, it is intended that features of one claim may also be included in any other independent claim, even if that claim is not directly made dependent on the independent claim.

[0058] Although specific embodiments have been illustrated and described in this specification, it will be understood by those of ordinary skill in the art that a variety of alternative and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present embodiments. This application is intended to cover any adaptations or variations of the embodiments discussed in this specification. Therefore, the embodiments are intended to be limited only by the claims and their equivalents.

Claims

[1] A method (100) for checking a machine learning-based model for an error, the method (100) comprising: Obtaining (110), for a plurality of samples, a concept activation vector (CAV) of the model for segmenting a respective sample; Obtaining (120) a respective ground truth segmentation for the samples; adapting (130) the model based on a comparison of the CAVs and the ground truth so that a deviation of a projection of the samples using the CAVs and the ground truth is reduced; Clustering (140) the CAVs into one or more clusters based on a semantics of the concepts of the CAVs; Obtaining (150) grouping rules for classifying objects based on the clusters; and Check (160) the model for errors based on the grouping rules. [2] The method (100) of claim 1, wherein the method (100) comprises: Obtaining the CAVs and the grouping rules for multiple layers of the model, where the grouping rules are layer-specific; and obtaining generalized grouping rules for multiple layers based on the layer-specific grouping rules. [3] The method (100) of claim 1 or 2, wherein the method (100) further comprises reducing a dimensionality of the CAVs. [4] The method (100) of any preceding claim, wherein verifying the model comprises verifying the model based on a comparison of the grouping rules and the further CAVs generated when applying the model to a test sample. [5] The method (100) of any preceding claim, wherein the method (100) further comprises adapting the model based on the identified error. [6] A machine learning-based model obtainable by the method (100) of claim 5. [7] Use of the machine learning-based model according to claim 6 for a driver assistance system and / or a system for (semi-)autonomous driving. [8] Facility (800), comprising: one or more communication interfaces (810); and a data processing circuit (820) configured to carry out the method (100) according to any one of claims 1 to 5 and / or to apply a machine learning-based model according to claim 6. [9] A vehicle comprising the device according to claim 8. [10] A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method (100) according to any one of claims 1 to 5 and / or to apply the machine learning-based model according to claim 6.

Images